How to set up AWS Kubernetes Jenkins pipeline? - Part 13

hello there this is rahul and in today's session we are gonna talk about how to set up your kubernetes jenkins pipeline on aws what is the end result of this exercise so at the end of this session you are gonna set up your kubernetes cluster running on a public ip address of your aws instance so here is the instance detail so this is my ec2 machine uh with the public ip address and on the same public iep address i have installed jenkins so this is going to be my jenkins server secondly we are going to set up kubernetes cluster on aws this is my cluster name and once you click on cluster then these are the nodes which we are gonna set up third we are gonna set up pipeline and we will start by cloning our code from a github repo then we are going to build and compile our code inside our jenkins server then we are going to build and tag our docker image and finally we are going to push our docker image to docker hub and then we are going to do the final deployment on our kubernetes cluster that we have already set up using eks before moving further let's discuss the prerequisites for setting up this pipeline on aws so the first prerequisite which i have is you need to have an aws account secondly you cannot set up kubernetes cluster and kubernetes services on aws free tire account you need to pay some prices but if you are doing it for a learning purpose then it won't cost you much moving further let's see how the complete flow is gonna look like so the in this flow the first component we are gonna talk about is the virtual machine since we are working on aws that is a cloud so we need to spin some virtual machine and that virtual machine we call it as ec2 and in this ec2 machine we are going to install some operating system so if you are going to work on aws so you need to choose an ec2 instance once you choose an easy to install then then you need to define like what kind of operating system you want to install in it so these primary things you need to specify when are trying to launch an ec2 instance that means we are going to start a virtual server okay so this is our first component where we are going to start our ec2 instance along with this session i am going to share this guide in this guide i have documented all the stuff which is i am going to perform in this session so if in case you want to check any of your previous step or the next steps then just jump on to this guide and see the instructions so we are just going to perform the same instruction into this session starting with the step number one the first step is we need to set up an aws ec2 instance so for that we need to go to our aws management console so here you can see this is my aws management console so once you set up your aws account then you should see this dashboard so here as soon after you log in you need to click over here and type in ec2 click on it and here on this dashboard you will see a launch instance option so here on this orange button you need to click launch instance click here and here uh the aws will provide you with the different options or the choices of the operating system which you want to use for setting up your virtual machine so we are going to choose ubuntu but in your case you can choose any other flavor if you want like suse linux red hat or amazon linux or even windows or is also available but uh i'm going to choose ubuntu and here uh you need to choose the instance type so here there are different options like t2 nano t2 micro t2 small and medium and then large so this is just an uh power of a cpu or the virtual machine so since we are going to set up or we are going to install jenkins on this machine so that's why we cannot use t2.micro because it is going to need some more power so that's why i'm going to use d2 uh medium then click on next configure instance detail and here in this step we are going to keep everything as default we are not going to update anything uh so simply click on next add storage and here uh if you need more storage then you can add it over here but since it's by default is eight gigs so i think that's self-sufficient enough for installing jenkins okay then click next add tags and this step is little optional but anyway we can add something over here so i can see jenkins ec2 yeah that should be enough i guess and then click next configure security group so here by default the option uh they have enabled its ssh 2022 port so that means you can ssh into your server on this port that is 22 uh but we are going to need one more port because we are going to install jenkins and that jenkins port is the jenkins is going to run on a port 8080 so i am just going to type the port 8080 and all this instruction you will find it over here so here we have gone through launch instance we have gone through like what to choose as an operating system we have chosen ubuntu then we have chosen t2.medium and then we have chosen like configure instance detail we kept it default 8 gigs of memory then we have added tag and then we have added this port also so which you can see over here that custom tcp port 8080 then click review and launch okay so it will give you some warning but uh you don't need to worry about it uh that's okay it's not an error it's just a warning uh it will say you like you are not eligible for free tire because we are using t2 medium so that's okay we know it and then we just need to click on launch okay so here uh this is an important thing uh whenever you are going to create an ec2 instance you need to create a key pair so this key pair is useful when you want to perform ssh or you want to log in from your local system into an aws machine okay so here uh i'm gonna choose uh i have already created some of my key pair so what i'll do i'll just create a new pair okay so i'm just gonna type the name my key pair 2 okay i'll click on download key pair and remember this key pair or this private this poem file you need to save it somewhere because we are going to need it to perform a login or to ssh into our server so i'll click ok and then we are going to click on launch instance it will take some time to launch your instance and you can simply go and click on this view instance button and here you can see uh the one which is pending uh that's the instance we have started okay uh this jenkins once uh i was working previously that's why it's there and it has been terminated but anyway we are looking on this instance which is right now in a pending state and here you can see now it's uh into running mode now just to keep things simple what i'll do i'll assign some useful name to it jenkins uh let's set two now we have completed the step number one and the next step is we want to connect to that virtual machine which we have just started now we have completed the step number one where we have uh launched an ec2 instance the second step is we need to connect to ec2 instance but before we connect if you remember we have downloaded a keypair file and the keypair file name is mykeypair2 and we need to use this keypair file to ssh or to login into our ec2 machine but before that we need to change some permission over here so the permission is since i'm using ubuntu as my operating system so i have this uh option like i can just go here and change the permission from here so but in case if you are not using ubuntu operating system then i would recommend you to use the command ch mode and here you can set the permission like 6 6 0 and then your key pair file name okay so this is how you're going to change it but anyway the permission i'm gonna show you over here so you can't allow other user to execute this and other you should set the permission as none they cannot read or write and similarly for group also you should keep it as none okay so this is the permission uh which you need to set on this file and then we are going to use this file to ssh into our ec2 machine now we have set the permission let's go back to our aws dashboard and here you can see we have set up jenkins 2 as our easy to install so click on this instance id and just remember like the instances that should be running so click on this instance id and here you will find an option to connect so simply go and click on connect and here you will find an option like ec2 instance connect session manager ssh client and easy to serial serial console we are going to use ssh client to connect to our ec2 machine so the command which we are going to use is this command ssh and this is the key pair okay so this is the key pair we are going to use it and you simply need to copy this command you don't need to modify anything so you just copy this command by clicking on this option go back to your terminal and remember uh you should be on to that directory where your keypair file is there so here i'm into this directory you as you can see and here i can locate my file that is my keypair2.pam so this is my key pair file so now i have copied the ssh command i'll just simply paste it over here so here it is going to use the same key pair file which we have downloaded and simply hit enter and here you can see this is the ubuntu virtual machine which i am able to log in using my key pair file i'll clear the screen okay so now if you check our step so we are done with step number one we have set up our ec2 machine we are able to connect our ec2 machine and the third step is we need to install the jdk so i'm just gonna click over here and here first of all uh what we gonna do we are gonna update the package manager since we have just started our ubuntu machine so we should update your our package manager so i'm just gonna copy this command and go back to my terminal which is over here i'm gonna paste it over here and hit enter so it might take some time but it should be quick enough to finish it up okay now it's done go back to guide and see uh what we need to do we need to install java so first of all check whether java has been installed or not so i'm just gonna copy this command and check if there is any installation of a java which has been done previously or not so i'll clear the screen i'll paste the command and here you can see java not found so that means java has not been installed so go back to this guide and if you haven't installed java before so simply copy this command that is going to install java and we are going to use the open jdk 11 version so go back to terminal and paste it over here so it is going to install jdk simply type yes and hit enter all right so now we have installed java i'll clear the screen and i'm just going to run the command java version again and here you can see we have installed java open jdk version 11. uh 0.11 moving back to our flow so what we have achieved so far as we have launched our ec2 machine after launching our ec2 machine we are able to ssh into our ec2 machine and after that we have installed jdk or the java onto our ec2 machine so these are the prerequisite which is needed for installing jenkins so that is going to be our next step so still we are talking about this first component which we are trying to prepare uh in this whole setup now the step number four is we need to install and set up jenkins so click on it so these are the instructions for installing jenkins so just copy the first command from here and go back to your terminal so this is my terminal where we have previously installed java i'll clear the screen and simply gonna paste the command over here and hit enter and after that we are going to copy the second command we are just simply adding the repository which is needed for installing jet gains so this is the second command for that hit enter and after that uh we are going to update our package manager because we have added the new repositories uh from where jetkins will be downloaded and installed on my operat and this ubuntu virtual machine so now i'm updating my package manager that's been done i'll clear the screen and go back to my guide and then we are going to install the check case so this is the command for installing jenkins simply paste it over here and type y and hit enter okay so now we can see our jenkin has been installed successfully and you can check the status of a jenkins by running the command sudo service jenkins status so copy the command go back to your terminal and paste the command over here and here you can see the status should be active now after installing jenkins we need to set up our jenkins also so there is a further steps also which is needed after performing the jet kits installation so go back to your aws console so here's the screenshot and where we need to click is the instance id so i will go to my ec2 click on instances running and here is my instance which is running right now click on it and here you will find the public ip address so copy this public ip address from here and the same instruction i have given over here also so this is the public ip address it will change for you because they are going to assign aws is going to assign a new ip address so simply paste it over here and if you remember i told you to enable the port 8080 tcp port 8080 in the step number one i guess so if you go back into the guide so here this is the port i have told you to enable that is 8080. so this is the port where jenkins will be running so now what i did is i copied my public ip address of my ec2 machine which is running inside my aws and this is my ip address of that i'll apply 8080 port after that and simply hit enter and this is the login screen or the start screen of our jetkins if you are able to see this screen that means you have installed jenkins successfully but to continue further you need to download the password or you need to get the password and the instructions is over here so you need to get the password from here so simply copy this path go back to your terminal uh clear the screen but you need to use admin privileges so you need to type sudo then cat for viewing the password and then just simply paste the command and here you can see this is the password which is generated by jenkins by default and this is a one-time password which is needed and after using it for the first time or that will be the last time and you will not be able to use it again so i'm simply gonna paste it over here and click continue uh i'm just gonna i'm just not gonna save the password and here customize change so i will go with the install suggested plugin option click here and it might take some time so i'll be back once everything is ready here you can see this is the next uh screen you will get where you need to create a admin user and you need to set the username password so simply i'm gonna type the username admin uh this is for demo purpose that's why i'm keeping the simple name but just for security reason use a strong password and username full name i'm just gonna type rahul and admin so these are the details i have fill in save and continue it will ask like would you want to run the jenkin on this url so simply say yes save and finish and start using jenkins simply click over here and here you can see this is the dashboard of our jenkins so which means we have installed our jenkins successfully and we are able to set up our first admin user on jetkins it sounds like we have done quite a lot but to summarize it we only set up our ec2 machine on aws and after setting up the ec2 machine we have installed jenkins so the next thing is we need to add some administrator privileges to the jenkins user so now i am talking about some jenkins user okay we have set up an admin user for jenkins that we are going to use to login inside our jenkins and to view the dashboard and perform some operation but there is one more user that is jenkins and once we run the pipeline so we are going to run the pipeline with the help of jenkins user so we need to add some administrator privileges to the jenkins user so that we can run our pipeline because in the pipeline we will be using docker we will be using gradle and then we are going to deploy all the artifact inside our kubernetes cluster okay so the next step is we need to add some administrator privileges to our champions user moving to the next step which is step number five we need to add some administrator privileges to the jenkins user so click on this link first of all and the file which we are going to add is our edit is etc sudo over so this is the file we need to update and in that file we are going to add this line so this line is going to tell like for the jenkins user don't ask the password because jenkins will have administrator privileges and we don't wanna type in those password again and again so that's why we are assigning this attribute no password okay so simply copy this command from here go to your terminal uh and paste it over here hit enter and you need to go into the edit mode and go at the bottom of the file sorry for this beep it's just coming from my laptop so here you need to paste that configuration so just copy from here go to your terminal and paste it over here press this key and then q a sorry it should be wq and semicolon okay so now we have updated our sudor file and now jenkins will have the administrator privileges moving further uh the whatever command we are going to run from now onwards so we are going to run as a jenkins user but before that we want to change the jenkins user and run it as a root user so that's why we are going to run this sudo su and of with the jenkins so i'm just going to copy this command from here go to my terminal and paste it over here so here you will see we are going to change the user and now you can see our jenkins user has been changed moving to the step number six we need to install docker so just click on it and here these are the instructions for installing the docker and just to give you a little brief we are going to create a docker image also inside our pipeline so we are just simply going to check out our code and after checking out our code we are going to prepare a docker image so that's why we need a docker that needs to be installed inside our jenkins server not inside the jenkins server but it needs to be insert installed parallely to jenkins we have installed jenkins and now we are going to install docker so copy this command from here go to your terminal and paste it over here and remember we are going to run every command with the jenkins user not as a ubuntu user okay because we have switched the user uh to the jenkins and jenkins has the jenkins having all the root uh privileges okay so now i'm gonna install docker uh simply type yes okay so now our docker has been installed let's go back to our guide and we need to verify the docker version so the command we are going to use is this docker dash dash version so it's i'm gonna simply copy it i'll clear the screen and here you can see this is the latest docker version we have installed it moving further the next step which we need to do is we need to add our jenkins user to the docker group because uh we are going to run the docker command so that's why our jenkins user should have the access to the docker group so that's why we are going to add our jenkins user to the docker group so i'm simply going to copy this command go back to my terminal and paste it over here and now we have added our jenkins user to the docker group one thing which i would like to point out over here is uh since we have added jenkins user to the docker group so we should be able to use the talker command so first of all just check uh are you able to run the docker image ls command or not so here you can see you got the issue like got permission denied while trying to connect to docker. so the simple solution to this error is you need to restart your ec2 instance and for that go back to your aws dashboard go to your ec2 instance click on the instances running select the instance and go to instance state and reboot click reboot so it is just simply going to restart and apply those changes which we have did like adding jenkins to the docker group and all and here you can see we have restarted our ec2 instance so our connection from the terminal has been closed so what we need to do is we need to relog in it so for that what you need to do is you just simply refresh it first of all click on this instance id and then go to connect over here and simply copy this command again and go back to your terminal and paste it over here again i think it's still restarting so i'm just gonna retry one more time and here you can see we are able to login so i'll clear the screen and remember uh we always need to perform any action inside our ec2 instance using jenkins so the command which we are going to run is sudo su jenkins so always remember we need to do everything with the jenkins user not as ubuntu or the root user okay i'll clear the screen over here now we have rebooted our ec2 instance so let's run the docker command again and see if that error is coming back or not so ideally it should not come back so i'm gonna run the command docker image ls and here you can see since we have not created any docker image that's why the list is completely empty but at least i'm not getting any error message like the docker diamond cannot be accessed so our jenkins user is able to access the docker after installing the docker along with the jenkins all right let's talk about our flow once again and see where we are so we have started with our first component and in the first component we have talked about ec2 machine so we have set up our ec2 machine that is up and running now we have installed jenkins on it we have configured the admin user we have configured the jenkins user and then we have installed a docker also and after installing the docker we are able to add the jenkins user to the docker group also so that jenkins user can run the docker command so that is all about our first component now what is next there are few more component that we need to install on this machine that is ec2 machine and the two components which i am talking about is aws cli and eksctl cli okay so why we need the aws cli since uh we will be using this machine to start our kubernetes cluster so that's why we need some kind of a mechanism so that we can authenticate uh this machine so that it can use the aws environment so that's why we need aws cli so what's going to happen is we are going to first of all install aws cli then we are going to configure the security key of aws cli so that this machine can execute the command on aws environment and after we have configured the aws cli then we are going to use the eks ctl so eks ctl is a cli for managing our kubernetes cluster so this is also command line interface and we are going to write some kubernetes command which is like provided by eksctl so that we can start our kubernetes cluster on aws environment so now we are going to install two more components that is aws cli so that we can communicate with the aws because it's still a virtual machine running somewhere but it to connect with the aws we need to provide that authentication mechanism so that's why we are going to install aws cli and then we are going to provide some security keys to it and then we are going to install eks ct let's go back to our guide and now we are into the step number seven where we need to install and set up the aws cli so click on this link and here this is the command which we are going to use to install aws cli so simply copy this command from here go to your terminal and clear the screen and always remember we are going to use the jenkins user to perform any kind of installation so just paste the command over here and this will install aws cli all right so now we have installed aws cli you can verify the installation of aws cli using the command aws version and go to your terminal copy this command and go to terminal and paste it over here and here you can see this is the version we have installed of aws cli i'll clear the screen moving further now we need to configure aws cli so what does it mean by configuring aws cli so we are going to provide two things one is access id key and the secret access key so that this machine can communicate with our aws environment with using this access key and secret access key and these are really private and associated with your account and also we need to provide the region so in which reason your aws uh clusters are okay so these are the three components which we need to provide copy the command aws configure from here go to your terminal and simply paste it over here and once you hit enter it is going to ask the access key id so what you need to do is go to your aws console and go to your username go to security credentials and if you have already then it's really okay if you don't have then this is the way you can create the security key and access key and then go to access keys and here you can see i already have couple of them so what i'm gonna do i'm just gonna first delete one and i'm just gonna create one so it will ask do you want first of all we need to deactivate and then i'm just going to copy it so that it will ask because do you really want to delete it so i'll just say delete okay so now i have deleted so i'm just going to create a new access key and remember this access key and secret access key is visible only once so once you close this then you will not be able to retrieve it so you need to recreate it okay so i'll just copy from here and i'll keep it somewhere on my notepad and simply i'm just going to copy this one also and paste it over here okay so now we are done with the access key id and secret access key i'm just going to close this and also we need to check the region so that you can check it from here by clicking on global and here we are using eu central frankfurt so here you can see this is the reason which we need to use but that depends in which uh part of the world you are so you need to choose based on that so i'm just gonna note this thing also over here let's go back to our terminal and then we are going to supply our access key so just copy from here paste it over here and then it is going to our secret access key so which we have already copied so copy it from here and paste it over here and then the reason so that is eu central 1 which currently i am in so i'm just going to paste it over here and then it is going to ask the default output format so you can just keep it blank and simply hit enter and this is how we configure our aws cli moving back to the guide now we have completely completed the step number seven the next step which we are going to perform is step number eight and where we are going to install coupe ctl so cube ctl is a command line interface for running the kubernetes command so that you can see what kind of a deployment what kind of a services which you have exposed inside your kubernetes cluster so that's why you need this utility so click on this link and these are the commands for installing cube ctl and remember we are installing everything on our ec2 machine where we have installed jenkin okay so we haven't started any kubernetes cluster yet we are still preparing everything so that we can start our kubernetes cluster in the later stages so just copy this command and install the cube ctl go back to your terminal and paste it over here hit enter okay that's done and then copy this command also it's just a simple command which you can find on the google documentation also because kubernetes is from google so the next thing is i just want to move this file to my current user so i'm just going to copy this and paste it over here hit enter okay that's been done i'll clear the screen and you can verify the installation of a cube ctl by running command q ctl get nodes so as you can see uh we haven't set up our kubernetes cluster that's why it doesn't show anything but that's okay we are gonna do it later i'm moving back to the guide the next step is step number nine and in that step we are going to install eksctl so this is the uh eka ctl is a command line interface for interacting with the aws kubernetes services so that's why we need it and since we have already installed aws cli so we have already authenticated or we have provided those authentication keys so that we can communicate with aws and we are going to perform everything on our ec2 instance so just click on this link and just copy this command this is the first command which you need to use for installing eksctl and you can find it from the official guide also in case if this installation command doesn't work then just go to eksctl official website and where you will find all these instructions there also so paste it over here hit enter that's been done and then run the second command where we are going to move the eks ctl to the current working directory of a user so paste it over here that's been done and then simply verify whether we have installed eks etl properly or not so simply are gonna run this eks ctl version come on and here we can see this is the current version which we are running for eksctl i hope you are copying with me there are lots of stuff which we have already did and there are still some stuff which we need to perform to set up our complete pipeline so till now uh we have finished our first component in our workflow and now our first component is ready our ec2 machine that is our virtual server is ready and we have all the required component installed and up and running so let's reiterate all the components so we have installed jenkins we have installed cubectl we have installed docker we have installed eksctl so these all are the tools which we need to setup or to run our kubernetes cluster on aws okay so the next thing is we are gonna use eksctl so i have told you ek ctl is from weave network and that's again an open source so we use this command line uh interface or command line framework to interact with our kubernetes services or the kubernetes cluster running in aws environment so we are going to write some commands using eksctl so that we can start our kubernetes cluster so we are not going to use the ui of our aws so we are just simply going to write a ekctl command where we are going to define our master and the worker nodes and then we are going to start our kubernetes cluster so that is going to be our next step and that is going to be our second component that is kubernetes cluster coming back to our guide till now we have completed our ninth step and the next step is we are going to create our kubernetes cluster inside our aws environment so that is our step number 10. so in this step we are going to run this command so this is this command is going to start a kubernetes cluster so if you go back to our aws so click on aws dashboard so you will see that there is no kubernetes cluster which is up and running so here you can click on elastic kubernetes services and click on clusters over here and here you can see there is no cluster although we can create a cluster from here but since uh we are talking about ci cd continuous integration and continuous deployment so we don't wanna do any kind of a manual work so that's why we are going to use the eks ctl to automate this kind of a stock okay so copy this command and run it on your terminal so i'm just gonna copy this command but before that i would like to explain this command so i have already copied this command over here so that you can see the breakdown of this complete command so this is my single line command but here you can see the same command i have broken it down so let's understand what this command does so this eksctl is a keyword and this is the command create cluster that's the keyword we are going to use for creating a kubernetes cluster then you need to specify the name of the cluster so here i have given the name as j hook test cluster third we need to specify the kubernetes version so i have specified the version 1.17 but you can specify any kubernetes word or any kubernetes latest version that's absolutely okay and up to you second we need to define the reason in which reason are you running your aws account then we need to define the node group name since uh you know we need to provision nodes inside our kubernetes cluster so we need to specify the node group name so here i have specified as a worker nodes then again the nodes are still a virtual machine so since as you remember in the first step we have started our ec2 machine that is t2 medium but here uh we need to assign what kind of a machine they are going to be inside for your kubernetes cluster so we need to define those nodes as a t2 micro since we are doing it for our demo and example purpose that's why i have kept t2 micro but you can keep t2 medium or large as as per your needs but just remember kubernetes cluster or the nodes are still a virtual machine that's why we need to specify t2 micro or the type of the machine which you want to uh start in your aws machine and how many nodes we we want so we want two nodes so this is the breakdown of our eks ctl command okay so simply copy this command from this guide and go back to your terminal clear the screen and paste it over here and simply hit enter oh i got the error but the error said this cluster already exist here you can see since uh before this recording i was just playing with this whole setup so that's why i have created this cluster already so what we are going to do we are just going to go back to our guide and we are gonna change the cluster name so i'm just gonna copy this and i'm just gonna paste it over here into my notepad and here we are gonna change the cluster to one okay and i'm just gonna copy this command and go back to my terminal i'll clear the screen and paste it over here and this is how we are going to provision or our start our kubernetes cluster and remember this provisioning or start of kubernetes cluster will take time uh for me it took around 20 to 25 minutes i have shared the screenshots also in the guide so here you can see it started around 1849 and it completed around 1908 so it roughly took around 20 to 25 minutes so be patient because it will take some time to start your kubernetes cluster but once your kubernetes cluster is started so you will see over here so once you need to you need to refresh it and then you will be able to see your kubernetes cluster so i will be back when this cluster is up and running now i'm back again and here you can see our kubernetes cluster has just started and it almost took uh i think more than uh 20 minutes and here you can see the timestamp when the kubernetes cluster is finally up and running so we can verify the kubernetes cluster so go to our aws console and just click on elastic kubernetes services and just click on clusters and here you can see j hook test cluster one remember we have kept change the name to cluster1 because a previous name was already taken by me and once you click on this then you will see the nodes also so here there are two nodes that is t2 both are t2 micro and these are the two node names which is available over here now we have set up our kubernetes cluster and if we talk about our flow so we our first component is ready and our second component that is our kubernetes cluster that is also up and running now before we move further uh we need to understand a one more flow like how our pi pipeline is gonna look like so i have divided our pipeline into number of stages so that we will start from a github and then we will end up on deploying our code or our container image of our application inside our kubernetes cluster so here this is the flow of our pipeline so here you can see this is the step number one and this is the code sign so that means this is my code and the first step which i would like to have it is like we need to push our code to github or any kind of a code repository so for this example i am taking github but you can push your code to either bitbucket github or svn so it needs to be in somewhere into the code repository so here you can see this is my code which i'm going to deploy inside my kubernetes cluster or we are going to run inside our pipeline so this is the code i'll share the link of this github rapport into the description section so you can clone this rapport okay so coming back to the flow over here so now we have pushed our code and that code reside in our github so here our code is residing on github now what we are going to do we are going to check out our code uh inside jenkins so that's the jenkins script we are gonna writing it so that's going to be our first step so jenkin wants to know your code so that it can compile and then proceed further so here if you go back to our guide so this is the stage so this is going to be a git clone stage so where we are going to clone our github rapo so here this is the url and as you can see ka jenkins aws so i have mentioned this is the my github url and here this is my github repo so this is going to be our first gen script where we are going to clone our uh github rapper of our application and it also need github credentials since it's public rapport so you don't need to provide your credential it will work uh without credential but just in case if it is a private repo then you need to add this github credential i'll show you how to add this credential once we are once we start writing our pipeline so this is the variable which we need to create and we need to store our password but i will explain it later so this is going to be our first step second step we need to build and compile our project since uh this application is a java application so that's why i have used the build tool as gradle but in case uh if you have a different application then this build tool will differ so if you have a node.js then you will not be able to use gradle so you will use something else similarly if you have any other angular or any other application other different framework then you are going to use some different build tool but remember this build step is simple yeah it just tells you like you need to compile your project so the compilation will differ in each kind of a framework whether it's a c plus plus a dot net application or angular or not js application so that depends but concepts are pretty much going to be the same okay so i can show you that one also so this is going to be our second stage that is built so i'm just going to run my compilation command so if this command is will going to differ for each framework as i already told you so just use that suitable compilation command over here since i'm using griddle that's why i have used griddle build command so now we have uh compiled our project and since we are working on a kubernetes so we need to have a docker image so what we need to do is we need to after compilation we need to build our docker image and remember we are doing everything inside jenkins so jenkins is doing itself once we trigger the jenkins build it is going to check out the code it is going to compile the code and then it is going to build a docker image so let's see what's how we are going to build the docker image so this is going to be our docker build stage so here what we are going to do uh here i am just checking the version what kind of a docker we have then we are going to build the docker image and tag it so this is the build and this slash d is for tag and i am going to tag it with the j hook docker demo that's the name of my image and here i am running the docker image list command uh so that i can see our image has been created so we will see all this output on our console of our jenkins pipeline okay so after we tag uh build and tag the image we need to tag it once more uh as per the name of our docker hub wrapper a docker hub repository so you need to have one more docker hub account uh there is i think uh aws container registry also but uh just to keep the thing simple you need to have a docker have repository also so that you can push your docker hub images so here uh if you go back to the flow what we have done we have compiled our code we are going to build our docker image and once we build our docker image then again we are going to push this image to the docker hub so here we are going to push our uh docker image so this is the wrapper name uh here you can see rahul walk j hook docker demo so this uh repository we are going to use for pushing our docker image so here uh first of all we are going to build and tag the image and the tag name should be the same as my wrapper name in my docker hub that which is this one okay so now we have build and created our docker image uh so the next thing which we are gonna do we are gonna push the docker image so we are going to use docker push command and we are going to push our docker image which we have built it over here okay so this is going to be our next stage of docker push so if you come back to the flow so here we are into this stage where we are going to push the docker image sorry next after pushing the docker image what we are going to do we are going to pull the docker image back again and why we need to pull that image back because we want to deploy inside our kubernetes cluster okay so go over here and here you can see uh this is the cube ctl apply f k8s springboard deployment yml and this 5ml you're going to find over here inside my code wrapper so once you're gonna clone over here in the step number two so you are gonna get all these files over here so if you go and see the content of this file then you will see this is the image name okay that we have pushed into docker hub so this is the docker hub repository so what's going to happen is ah we are going to apply this configuration this configuration kts springboard deployment yml this we i have already written you can modify with your uh image name so this is the docker image name so what gonna do we are gonna just simply apply this configuration so here if you go to the go and see the guide so here i'm writing cube city will apply and the yml name this complete yml name and this yml contains the docker image name okay so that's why in the flow i have mentioned like we need to pull the docker image so once you are going to uh run this stage that is cube ctl apply it is going to first of all pull the docker image and after that pulling it is going to deploy the docker image and everything is done via this configuration so this is the deployment part and after deployment this is the service part so it is going to pull this docker image it is going to deploy inside the docker container as inside the kubernetes cluster and then it is going to expose that service on a load balancer so this is the complete uh flow of our pipeline and all the scripts i have already mentioned over here so we are this is the complete script this is going to be our final script here you can see okay so again i'm gonna reiterate it so this is the git clone which we need to do then we are going to do the gradle build then we are going to create a docker build where we are going to create a docker image and we are going to tag it and then we are going to push our image to the docker hub and then we are going here sorry here it is here we are going to push the docker image to the docker hub this is the docker hub credential because you need to provide the username and password to push the docker image and finally we are going to deploy our container or the docker image inside our kubernetes cluster as i told you we need to store the secrets uh and for that we are going to store the secrets or the username password of github as well as dockerhub so these are the two credentials which we need to store inside our jetkins so go to your jenkins so this is my jenkins it is up and running and i'm just gonna use the admin which we have set as a preferred username and go to manage jenkins go to manage credentials and here click on add credentials and just follow the guide and first of all what we are gonna do we are gonna add the docker hub credential so id we are gonna use is first of all the kind we are gonna use its secret text so here uh go into that and go secret text and this is going to be my password of docker hub so you need to type in your password over here of your docker hub account so i'm just gonna put my docker up credential after that you need to assign the id so i am assigning the id docker hub password but it's up to you what kind of a name you want to keep so but just doing no you need to put some id which is unique so i'm just gonna put docker herb password and remember we are going to use this id letter and description you can put anything docker password that doesn't matter click ok and again we are going to add one more credential and the next credential we are gonna add is for github so this time you are gonna take username with password and uh you can go over here username with password so username i'm just gonna take it from here rahul vog so it's just simply type the rockwool block and the password of your github account and the id again and the id should be unique and you should remember it because we are going to use in pipeline so this is github credential and if you remember this is the same keyword we are going to use so just keep the same thing all over the place okay so get up credential you can put description of your choice it doesn't matter what you're gonna put but put some meaningful name over here click okay okay so now we have a docker password as well as github username password there again go back to your dashboard uh click on new item and here just type test pipeline click on this pipeline option click ok and then go down further and there is one more thing which i forget to tell you we need to set up gradle also so go to manage jenkins and go to i think it's the global tool configuration and here we are going to choose gradle because since it's a springboot application that's why we are going to choose gradle so default and yeah just apply and click save okay so that was the thing which i missed to tell you again go back to dashboard and i i have already created the test pipeline so it is over here so again if you forget that step then simply type any pipeline name and select this option so it will create a pipeline for you okay so i'm assuming that you have created this pipeline go to this pipeline go to configure and go down and here we are gonna write all the scripts which i have mentioned so i'm gonna increase the font over here so that you can see and what i'm gonna do i'm just gonna go back to my guide and copy all this thing from here because these are the scripts we are gonna need it copy go to your pipeline and paste it over here so here again its name git clone using gradle build build your gradle project then here we are going to build our docker image and then we are using the docker of username password and using the same username password we are pushing the docker image and then finally we are doing the kubernetes deployment so apply and save it now our pipeline is ready and we can just simply click on build now and then you will see a number of our build that is one so click on it and here you can see our build is going on so i what i'll do i'll it will be better to see in a little ui mode so here you can see all the stages so first of all it did the git clone now it is doing the gradle build so it is compiling and building our project okay so now the gradle build has finished in 45 second and now it is going for docker build so that means it is going to build and tag our docker image and since now our docker build image is done so it is going to push our docker image to the docker hub and finally it is going to perform the kubernetes deployment okay and here you can see the our pipeline has went green there is no deployment failure and here the same thing you can verify from here also and go to console uh and just one more time i can show you so this is the our first build number which we got because this is going to be our first build and just click on it go to console output and i can show you all the outputs over here also so here uh we have done the git cloning after that we have build our or compile our code and then after compilation uh what we did is we just build a push build and push our docker image so this is the build of our docker image then here we are going to push our docker image somewhere here it should be here so we are going to push our docker image and then we are going to finally do the deployment and our final status should be success okay so now our pipeline has finished and we are able to check out our code we are able to build compile and deploy on our quality cluster now we have successfully built and deployed our first pipeline so we need to verify our deployment inside our kubernetes cluster so we can verify our kubernetes deployment using the command cube ctl getdeploy and remember we have installed the cubectl on our ec2 machine so go back to the terminal so here i'm just going to use my terminal i'll clear the screen over here and the command which i'm going to run is q cdl get deployment and here you can see my spring boot application has been deployed inside my kubernetes cluster secondly i just want to check the services whether our kubernetes deployment has exposed the services or not so for that the command is cube ctl get service and here you can see i'll increase the terminal size over here so that you can see in a single line hopefully it should be okay now so this is my springboard service of my deployment and that is running on load balancer and this is my external ip address which i got okay so i hopefully should be able to access my rest and point and the rest endpoint which i am talking about is hello world so here you can see this is my hello world message which i should get back once i execute my restaurant point using the public ip address or the external line what i'm going to do i'm just going to copy this external ip from here go to your browser paste it over here and remember our rest end point is hello so here you can see this is my rest and point hello so what you need to do is you just need to put in your hello over here and hit enter and here you can see uh this is the message which i got so hello j hook akk it is i have updated the message so this is how you are going to deploy and build your first pipeline on aws using kubernetes cluster and yeah i have used the springboard application but you can use any application of your choice all the concepts and all the steps are going to be same only the step which going to defer is how you're going to compile your project so this is gradle so this step is going to be differ differ i'm going to be different for you so just choose an application create some hello world up example and then try this complete pipeline setup on aws i hope you like the today's session on how to set up our kubernetes pipeline on aws environment and i know this was a long awaited video which i was planning to shoot but then uh yeah i just wanted to keep the things simple that's why i have divided uh the stuff in which i can keep the things very simple and bare minimum so if you have any question related to this session then please put down into the comment section and i'll try to answer those and for future uh i will i'm planning to prepare something more automation like here we are still trying to create a ec2 machine manually by going into aws but in future i am planning to create some session like where i will be using terraform and ansible and along with our kubernetes and jenkins and pipeline so there will be more automation we will try to do on the similar things which we have done today but yeah let's see uh what uh what ideas i got and what session i prepare in future and also there is a helm chart also we can do the similar things using helm chart also so that's also in my mind so i'm not sure what gonna be the next but it will be something around the helm chart terraform ansible and similar ansible sorry kubernetes and pipelines if you enjoyed this video then please do like this and subscribe to this channel and i keep on posting similar content on this on my channel on a weekly basis and you can find me on a web also with the name jhoop.com where i prepared all this kind of guides which is really handy when you are trying and when you are working on aws or cloud environment and trying to perform the similar things so those guides are really handy for you so i'll put the link of those guides into the description section also so yeah do share with others and let me know uh what do you feel about this session and if you have any feedback then those are also welcome so see you into the next session till then bye bye

Channel: Rahul Wagh

Views: 1,359

Rating: 5 out of 5

Keywords: aws, kubernetes, pipeline, aws kubernetes jenkins pipeline, devops, k8s

Id: ENNBdmNn4r0

Length: 58min 50sec (3530 seconds)

Published: Mon Jul 12 2021