Gitlab CI CD Pipeline Tutorial [ FULL COURSE ]

[Music] before understanding gitlab ci cd let's quickly see what exactly is ci cd let's say you have a developer who performs code commit on one of the feature branch a pull request is then been created for the code merge once it is approved the code is merged to the deployment branch which triggers the build creation using tools such as maven gradle npm once the build is created it triggers the test cases using tools such as selenium junit and the test cases gets uploaded to the required tool then it triggers a security vulnerability checks along with the static code analysis using tools such as black duck sonar cube then the bills get uploaded to a build repository such as sona type nexus jfrog artifact this completes the process continuous integration and to integrate this together we have tools such as jenkins aws code pipeline gitlab ci then as part of deployment it triggers the deployment to the dev server once the sanity is passed the deployment gets triggered for the uad server once it is reviewed and the sanity gets passed then that broad deployment gets triggered and once that is passed this complete process is completed which is called continuous deployment or delivery and it can be done with the tools like ansible helm aws code pipeline the difference between continuous deployment and delivery is continuous deployment needs some manual trigger before the product deployment while continuous delivery is an automated part of the deployment life cycle it is followed by continuous feedback and monitoring at all the stages build tests scan upload dev deployment uat deployment and productive learning this completes one release cycle but this is a never ending and the continuous process in this section we're gonna see a quick introduction of gitlab ci cd what exactly it is so i'll go to gitlab.com where i have already created an account and then over here i can see i have three different projects created so i'll go to this one uh the project name is ci cd as it says and then this is the root page this is the main page of the project where i can see the overview of my project with all the source code that i have currently i only have uploaded one file which is shown over here and then it also has a repository section over here where we could see all the files comments uh branches tags basically uh all the uh you know our source code versioning maintained over here then we have issues over here which is like a jira board then uh we can see merge requests all the pull requests or more merge requests whatever you want to call it uh it will be visible over here all the closed ones all the open requests conflicts resolve everything would be here then we have ci cd where our ci and cd pipelines will decide so we have different options we could see a pipeline editor or we could use the sample templates that have already been created by gitlab and then we also have some jobs schedules and test cases for running the test cases jobs and schedules are part of pipelines itself so i'll walk you through when we uh when we are there then we have uh packages and registries over here so this is basically a build repository for your packages like jfrog artifactory or sona type nexus so here you could upload your packages in different formats or you could upload your container images as well if you are running a kubernetes based application or a docker based application then we have a wiki over here which is similar to what we have in confluence so this uh complete uh ci cd platform ci cd platform provided by git lab is a complete suit that have all the application needed to perform your continuous integration and continuous deployment and delivery it has everything in one place so so you don't have to worry about integrating different tools with each other and to establish communication between them so all the pain is no longer there all the important settings related to cicd project within gitlab would reside over here if you hover your mouse to settings tab you'll see different options so you click on ci cd and uh so there are like a lot of settings over here but we'll be looking into few of them uh in in detail in the next upcoming sections and uh so for example this one run us over here if you click on expand it'll show you different uh options for runner so we'll see what a runner is how to install it how to use it and how to manage it then we have artifacts so this is basically just a checkbox which says keep your recent artifacts from successful jobs yes or no then we have variables so you could add protected variables like some secret keys and some mask variables as well you could just you know simply click on add variable and provide the key value here that can be used inside a pipeline then we have pipeline triggers which are like web books or the api calls which would trigger the pipeline then we have automatic deployment rollbacks so if you know he says if the if the build is getting failed or if it's not successful then roll back to the uh to the recent stage so we'll just keep it default for now so for our next topic let's go ahead and create a very basic uh pipeline in a simple project so i'll click on new project over here and then i'll choose this one create blank repository and project i'll give this a name let's say my first project and i'll keep this as private click on create project this will create a sample project for me without any files in it so i just have to set up i just have to show you the basic how a basic ci pipeline looks like in gitlab and what are the important points to remember in that so for that i'll go over here ci cd and click on editor and then it'll open a basic pipeline for me once i click create new ci cd pipeline so it has generated a yaml for me with few stages and few jobs so every project in gitlab can have a maximum of one gitlab hyphen ci dot yaml file this is basically your pipeline as a code and all the instructions of your ci cd written and and this in the yaml format right so whenever you upload your pipeline code to the project the gitlab detects there has been a pipeline uploaded to the project so it will trigger the pipeline execution for you you don't have to manually trigger it this is how ci cd works and once the pipeline is uploaded to your project and any changes in the particular branch or repository or as per the condition specified in the yml it will start executing your pipeline okay so with that being said uh let's see what does it have so it has see three different stages a typical uh ci cd pipeline will have these three stages build test and deploy so every stage will have one or more jobs so let's say the first stage is built it has a build job okay and has a script script will have your task written in the preferred choice of your language then the next stage is test which has unit test job and lin test job so this test stage has two different jobs then we have a deploy stage which has a deploy test job okay so if this is too much for you don't worry about it we have different options to make it uh more clear to you so if you click over here visualize this will show you what i'm talking about so build stage will have build job test stage will have these two jobs deploy stage will have deploy jobs okay so it would have made more sense to you now that you have seen it visually okay the other important thing to remember about jobs and stages is all the jobs inside one single stage will be executed parallelly okay so let's say the pipeline gets triggered first it will trigger the build stage it starts with the first job which is build job then it will trigger the next stage which is test stage in that test stage it will trigger these two jobs in parallel then it will go and trigger deploy stage and this stage it will trigger deploy a job so i'm just gonna click comment changes over here and as soon as i commit the changes it triggered the pipeline says checking pipeline status and it's you know checking the agents in the background so it says pipeline this this id is running for this particular commit id okay so we can check the pipeline status from here if we open this is new tab or we could just go from the from the pipeline tab over here let's open this in a new tab and this is all the information that we have uh status is running this is the pipeline a unique id generated by the pipeline triggered it was triggered by myself by a git commit it was triggered on the main branch as we only have one branch and this was the commit id which triggers this job and then this was the comment which was like auto generated and these are the stages so first stage is running it's been passed now and then second stage is running now which is test the stages deploy duration is in progress and you can cancel the build from here as well so if we click over here and at the running we'll see uh the visual representation of the real-time pipeline status so we see the build has been passed and all the two jobs inside the test stage are running in parallel that's why you see two timers over here and it will be completed soon so let's click on this job which has been completed and you'll see all the locks over here so it says running on gitlab runner uh so we'll discuss gitlab runner in detail in the next uh few sections so it did all those things that we have uh you know written inside the yaml code it basically had some echo statements and then it says job succeeded if we go back let's go back to the pipeline again ci cd and then pipeline okay i'll click on this one okay so the first job has been completed if we go to the second stage it has two different jobs so if you click any one of them again it will show the same thing it will show all the locks that have been generated with that particular job and it says job succeeded at the end and the status is passed same way uh the second uh second job must have been triggered okay so it says uh now that all the stages are completed it says the status is passed so this is the status of overall pipeline it took 18 seconds from you know start to completion now it was you know triggered 20 seconds ago but it took around two minutes from start to the completion okay and we see all the stages have been there has been a you know green tick in each of the job if it would have been a failure in any of the job it would have shown you a red cross in place of this one okay so i hope this would have made more sense now that you have seen it uh visually so this was just a test pipeline we haven't created any you know any complex script inside those stages it's not even a real world pipeline but it was just to give you an overview of what's what we'll be dealing with next so in next section we'll be seeing uh how we can create a project a sample pipeline with with a sample code and we'll deploy that code in one of the servers after building and testing it all right so let's jump into the next topic now which is kit lab runner so let's quickly see what's a gift lab runner and how many types of runners are there so gitlab runner is an application that runs job in a pipeline can be installed on your cloud offerings or an on-prem infrastructure can be installed on many different operating systems you can also run it in a shell a docker container or a kubernetes cluster but exactly the process is so devops person registers that agent with a token and then it established a trust relationship between a gitlab instance and the communication channels will be open let's see how many types of runner are there so first is a private runner which is like a self-hosted agent and can be installed on your own data center or your cloud services the next one is a shared runner which uses the runner from the available pool of runners from gitlab based on the tags the next one is the group runner which can be installed on a kubernetes cluster and supports high availability and fault tolerant all right let's just uh start the demo then okay so i have my uh project created over here in gitlab i have created index.js file a gitlab ci ml file which is our pipeline as a code and a package.json which was generated as part of our local npm initialization let's see what's there in these files so i'll open the files in my editor so let's just see first index.jx this is just a sample hello world sort of javascript where we have initialized that it will require express npm module it will run on port 3000 and we'll just print hello world to the to the web page and it'll log this message in the in the logs okay and uh then we have a gitlab. then we have package.json this was created when we initialized the npm module and so it has two scripts test and uh start test script will just have a curse statement and then it will just exit with a success message and then let's see our gitlab ci which is uh but which is our important one so it has three stages so this is the same pipeline that we have used earlier in this demo it's just i have added a lot more details to it so that we could test the complete end-to-end ci cd with the hello world application it has those three stages build test and deploy and we have used cache over here with the path has not moduled so cache is it can be initialized globally with this key value pair so we use key and the value as a ci commit ref slug which would make this cache available throughout the pipeline in all the stages and all the jobs uh throughout the project okay so that we don't have to uh download or upload node modules over and over again and as it will have so many files and it will definitely hit the performance of your pipeline first job is build job which would be running inside a build stage it will use the image as node.js with the latest stack and it's part of a shared runner so we have specified runner over here so it will done this particular job will run where runners tag is docker which is our shared runner and it will use the image as node latest so that we don't have to install node.js on top of the image that we are using then uh these are the script that will run inside this particular job it will echo this one then it will do a git clone on on from a git repository on the develop branch it will go inside the project and it will do a npm install express module and then it will go back a directory it will do an apt-get update and then it will install the zip utility for us then once that has been done once the code is being cloned in npm is installed then it will zip the entire folder with exception to these two modules because no modules we have we are using it as a cache so we don't have to download it or upload it as part of build it will be available throughout the project to use and dot git would have certain git configuration uh you know header add details all those things get locks all those things that we wouldn't want as part of our build so we have excluded these two folders with hyphen x and so after this step a zip would be created with this particular name okay and it will be stored in the pipeline as an artifact because we have specified it over here use this as an artifact and path is uh the you know the home directory of the of the runner so whichever runner we are running is part of this particular pipeline in the home directory this artifacts would be available to use okay and uh so this we have seen like this particular job will be running on a shared runner with tag as docker and this will only trigger when a change has been made to the develop branch it doesn't matter like what sort of change you are doing whether the changes in the code or within the pipeline itself this stage will trigger this job will also be triggered on this particular develop branch then these two changes are same like what we have seen earlier so i'm not gonna go over uh this again unit test and lint test job uh this is just this is just i have kept it there to show you how us different jobs would run in parallel in a particular stage okay then uh this is kind of an important one which is deployed dev job this is part of stage deploy and it will run in my private runner which is my gitlab runner that we have that we're gonna see how we're gonna install it first step is because we would want uh our use case is we would want our artifacts to be transferred from our gitlab runner to our uh to our ec2 server right so we have to set up a ssh connection between gitlab from gitlab to a c2 server so we'll first see if ssh agent is present on the runner or not if not it will do an apt-get update and install the openssh client then it will create a dot ssh directory in the user's home directory it will start the agent as a search agent on the on the gitlab runner then it will echo the private ssh key and add it to the ssh you know keygen utility right so this particular variable we have used from the from the ci cd settings only so we'll go back over here we go to settings and then we go to ci cd so we created a variable over here if we expand variables field so we have created a private ssh key variable over here this value is uh is the value of our private ssh key that we use to login to our ec2 server so this is like dot peak dot ppk file okay make sure you use the value of dot p p k file not the dot p e m file okay and uh then we're gonna add this line strictos key checking as no so that it won't prompt you to add the key in the verified host so we add this to the ssh config and then we also add the dev servers public ip to ssh known host again for dev server this is the public ip of the ec2 server of the dev ec2 server that we have created over here okay and then it will so once the connection is established once the ssh setting is done now we have a passwordless connection from our gitlab runner to our ec2 server then we transfer the files then we transfer the artifact from our gitlab runner to the ec2 server so remember we have used this uh myproject.zip as an artifact in the first stage in the first build stage so it is available uh in the runner runner's home directory so we could just use scp hyphen r current directory and the name of the artifact and then the destination would be ubuntu is the destination username and other dev server which is the public ip of our dev series server and this is the destination directory okay after this step artifact would be transferred to the ec2 server now we have to deploy that artifacts so we'll do a ssh to the dev server ubuntu at the right dev server and we'll be running all those scripts as part of the deployment so we'll first have to do the unzip of that particular package okay and uh we have to make sure that zip utility is available uh in in that ec2 server okay so we'll unzip it and then we'll go to that particular package package directory and we'll do an npm install express hyphen iphone save we don't have to uh you know we did not transfer the node modules to that particular directory because we have used it as a cache so it will be available by default and then we use forever stop index.js to stop the already existing running application if it's already been running so forever as we have discussed earlier as well forever is a npm utility that we have uh used to run the process in the background okay otherwise if you would have run npm build or npm start it runs interactively and it run in the as a foreground process okay so we'll stop already existing uh index.js if it's there and if it's not there it won't throw you the error because we have used pipe pipe true okay it will uh you know suppress that error for us and then it will start the new application so this is all like what we have to do as part of deployment okay so even because this is just a node.js project we don't have to use it use the build we could just do a git clone and npm uh install in in all the servers it just like because i have to show you the difference between cache and artifacts and how we can use that in an actual project so that's why we are using it like that okay then it will uh run this process in my runner which is my private gitlab runner okay and it'll trigger only on the develop branch okay that's it so i'll just uh save this now as part of next step we'll be creating two ec2 servers one for get lab runner and one for our actual dev server and we're gonna do some installations on top of that so let's just head over to our aws console and create uh two ec2 servers so i'll go back over here okay and i'll click on launch instances and then i'll select a free tier ubuntu server from here i'll select e2micro and then i would need just one of those so i'll keep it uh one and rest of the settings as default over here we'll be adding some user script because we would want some uh installables to be run once the server is booted up so i'll just uh add this user script section over there so let's just quickly see what it is so this is like a bash script which would just install the runner the gitlab runner this is for get first we are doing it for the gitlab runner so this will install a gitlab runner then it will install the node.js and it will also install a npm forever package that that we have used inside the pipeline then on top of that install docker and uh it will provide permission to the docker pid and docker sock so that it will be uh we can we can run it with the ubuntu user without without using a pseudo command so i'll just copy the script from here and i'll paste it in the user data section over here it will verify if everything's good and yeah it's it looks good i don't need this one so i'll just remove this one okay and i'll click on next add storage i'll keep the default storage and we'll add tag i'll add i'll give it a name so name would be get lab runner so i'll do a good lab runner okay and i will add a security group so i'll create a new one it says get lab ssh group okay so we would want uh to do ssh in that server so i'll just keep the ssh port open so generally you would restrict it to your ip or or a cidr range behind your corporate network but because i'm gonna just decommission these servers after this demo so i'll just keep it open for everything so it will open the inbound and outbound rule for ssh to this particular server which is what we wanted click on revenge launch launch and then i'll use a key pair i'll use this one knowledge and i click on launch instances okay while this is being provisioned i'll create one more instance for our dev server so i'll click on launch instances again and uh then we have ubuntu image i'll select it from here t2 micro this should be good configure instance details okay rest of the settings would be same and for user script data section so we don't want all the all the steps to be performed over here because we don't want to get labrunner over here so we would just want to install node.js package and that should be enough for to act as an ec2 server so i'll just copy this one from here or maybe i'll just yeah copy and paste it over here and then i'll remove the steps for gitlab runner okay and then i would also want to install our zip utility so i'll do a sudo apt-get update hyphen buy and after this i'll do sudo apt-get install zip hyphen y should be good then i'll click on uh next add storage i'll use the default storage i'll give it a name name my first or i'll give it dev server configure security groups now i'll create a new security group dev server okay so i would want to do ssh in that group so i'll keep this one open and then i would want our gitlab server to do ssh in that so we have this open for everything so this should be enough but you could specify your ip of your gitlab runner over here or and your private ip i mean your personal ip from where you are logged in then i would want this particular server to serve the web pages over http and https so i'll open those ports as well https and http over here then uh our forever utility or our node.js package will be servantport3000 so i'll open a custom port as well okay and that should be good now i'll click on review and launch oop i haven't specified the name yet so i'll do anywhere then review and launch launch and i'll use the key pair launch instances and that should be good now that our gitlab runner ec2 server is up and running so i'll just open ssh connection with that server so i'll click on connect and i'll use the username as ubuntu i'll use this one okay i'm logged in so let's see if the user data script executed successfully let's see if our docker was installed so i'll do a docker hyphen v and it shows me the docker version that means it's been installed let's see if our gitlab runner was installed so i see the help commands over here so this is also been installed and let's see for npm installed and yeah it's already there so we have all our packages installed correctly which is what is expected out of a user script so let's just go ahead and and register this runner so i'll do a sudo gitlab runner register okay it says enter the gitlab instance url so i'm using a public gitlab instance so for me it will be the same https getlab.com enter the registration token so registration token you will see in the gitlab so you go back to your ci cd settings okay and you expand the runner you'll see available specific runner over here this is my runner that's already been running so i'll stop this one first okay and i'll so it'll show me uh the setup instructions of installing of registering a new runner so it says install gitlab runner and it shows it's running so this step we have already done as part of user data script then it says register the runner with this url this url we have already pasted it over there and then it says the registration token so i'll copy this registration token from here and i'll go back to my ec2 server i'll paste the registration token over here okay i'll hit enter then it says enter a description for runners i'll just say my runner enter tags so this tag what we'll be using in our in our gitlab ci pipeline so i'll give it the same name as we have specified in a gitlab pipeline which is my runner hit enter and then it says registered runner now we have to choose an executor so because we have installed a docker on top of our gitlab runner machine and the best way to use it is the docker executor so i'll just give it docker so these are all the available executors that you could use so i'll be running as part of a docker runner so i'll hit enter enter the default docker image so i'll choose node latest okay and it says uh runner registered successfully now we have to start it but before starting it we also have to create this directory over here else it will fail so i'll just create this directory where libritrap runner says permission denied i'll use the pseudo command with it okay now i'll run get lab runner start it says uh not supported for non-duty users so i'll use a sudo command with it as well and it's showing me the runtime platform it might have been start let's go back to our ci cd and refresh the page and we see it's running over here the status is green it says runner is online last contact was one minute ago and this is an id generated for the runner this is a runner stack so our runner has been installed successfully and registered successfully with one of the gitlab servers so now let's go back over here now let's login to our dev server click on connect open the session and i would just wanted to verify whether our user scripts ran successfully so remember we installed a zip utility let's see if it's been installed it says if not found that means it's not been installed there might be some issues with the user script so what i'll do is i'll do a sudo apt-get update hyphen y it's done and then i'll do a sudo apt-get install zip hyphen y and now it will install my zip utility it's good that we have checked uh in the beginning else our pipeline would have failed and we had to do it this all over again so now it's been installed let's clear this okay so we have verified this as well now let's uh go and see the next steps now that we have done all the prerequisites so let's just do one more thing let's just go back to our instances go to dev server i'll grab the public ip v4 address from here and i'll go to my cd settings so ci cd settings and then i'll expand the variables which is over here and i'll just update the dev server variable over here because this was the ip that was that i used earlier and now i have a new dev server so i just have to update the ip over here do that and click on update variable so we have our private ssh key just make sure you are using the ppk key of your ec2 server not the pem key or any other key it's a ppk private key that you use to log in your ec2 dev server and then this is the public ip of your dev server so these were all the variables that we have used inside our gitlab ci pipeline i guess this should be good so now just let do a save stage the changes stage all i'll just give it a commit id starting the pipeline i'll commit it to the develop branch so this is my local develop branch and i'll push it to the remote develop branch which would be from here and as soon as the changes are pushed to the remote develop branch it would have triggered the pipeline automatically so let's go and see if it did so i'll go to ci cd pipelines over here kick the pipeline for us so i'll click on the running and these are all the stages that are part of pipeline so first is our build job which is in progress so i'll click on that it will show us a real time log of that so because we have used a docker based shear runner so that's why it's preparing it and it's running gitlab runner on that with the image as node latest so i'm not gonna pause the video for this because i want you to see it in real time so we'll just give it a few seconds because it's uh you know shared runner so it will take some time okay so that's what it's been doing so all the commands ran successfully job is succeeded let's quickly verify if there are any errors so it says uh evaluating this which was for cache then it initialized a git repository it created the cache it downloaded the cache okay then it compiled the code and then it cloned the first project it cloned the repository with git clone command did an ad apt-get update installed the zip utility and then it ran the zip command successfully it created a zip with all these files then it uploaded the cache to a temporary location to be used anywhere else in the pipeline and then it says uploading artifacts which is our build let's go and see the other jobs so i'll click on pipeline again and so our test cases has also been passed let's go to the deploy stage which is in running and now it says it's it's running on my runner with this id so this is the runner that we have created on our ec2 server okay it's using the docker executor with image node latest uh we could have not used the image as well in this because we have explicitly installed node on our gitlab runner so if we go back over here oh i don't see i have used the image over here but somehow it took the image because that might be the default image that it had used so it's fine for now so meanwhile this is in progress let's go ahead and see the other test jobs oh it's been passed successfully so it might not have been refreshed okay so let's see so it says running on this runner which is our private runner then it says checking cachet for development one instead a local cache will be extracted it downloaded the artifact it downloaded the cache okay and that's been done it ran all the commands successfully it said identity added and then it scp the file the artifact from the gitlab runner to the dev server successfully i don't see any error with that then it did the ss to the dev server and run all the commands so cd first project and then it did an npm install i just see warnings over here but i don't see any actual error so it had installed the npm express package as well and uh then because we have uh used forever stop index.js and there was no index.js already running that's why it uh it threw the error but then we have used the true command with it it showed us the error but it did not fail the pipeline and then uh we did a forever start on index.js so it was successfully and then the cache created job succeeded so let's see if pipeline did the actual deployment so if everything worked fine then we should have with that web page up and running so i'll copy the public ipv4 dns from here paste it in the browser with the port 3000 because this is where we are hosting it on and i see my application is there this is what we wanted to do okay so it worked perfectly for us it worked as expected now let's go back to our pipeline i want you to see few more things in that so if we see the pipeline the status is passed all the stages have been passed successfully it took total of two minutes it triggered on the develop branch with this particular commit id and then if we click on these three dots over here it will give you the artifacts to download because i said like there's a difference between cache and artifacts cache cannot be downloaded externally or it will just be you know accessed from the pipeline and you cannot uh you know do anything else with that but artifacts can be shared it can be download it can be shared between multiple projects as well so you could just you know click on that and download artifacts and it will download that zip folder for you okay so we see all the stages that has been passed successfully all the jobs have been passed you can see the time minutes for each and everything okay so in this section of the video we'll see the difference between script before script and after script and how we can use it effectively inside a gitlab ci pipeline okay so we'll start with the before script so before script can be before or after scripts can be you know global to the pipeline let's say we define it over here with this syntax before script and then we uh you know provide echo statement this is before script okay and then we have after script we can define this here as well globally okay so what it will do is uh when the pipeline triggers when it starts it will see what's there in the global before script and it will run this command before the starting of every job where the before script is not defined so right now we don't have any before script section in this in this pipeline so it will run this command before every build job but let's say if we put the statement inside a particular job let's say this one over here and we call it this is bill job okay so now that we have initialized this inside a build job so this before script will override the existing one okay and the rest of the jobs will still run this before script okay so this will run before the job this will run before every job okay and this will have the higher precedence so i'll just remove these two and as the name suggests before script which run before the actual script and after a script which run after the actual script so i'll remove this these two okay so i'll just keep this before script over here and maybe i'll use this statement okay i'll remove this one and same way i'll use this statement as after script okay this should be good now let's go to the script so script is like we can define all the commands in sequence like this as an array or we could do simply uh we could put just a pipe and everything will be part of one single script so what we'll do is to make the code more readable you know for the sake of simplicity and easy to maintain the pipeline code we'll just copy everything inside the main script and create one more file inside our project's root directory so i'll just go over here i create a new file we'll call it build script dot sh okay and then i'll just start a new shell script over here and i'll put everything over here i'll just remove these because now it's a shell script everything looks good and then uh so i'll just instead of using all the statements over here i'll delete this one and i'll just call bash build underscore script so instead of you know specifying all the commands over here we could just uh wrap all the commands inside a shell script put it in the project's home directory and then we could just uh execute it with the bash command over here okay so this should be good so the same way i'll keep these ones as default for the same uh with the same way i'll just go over here i'll create a before script over here okay and i'll put all these commands like which which are like pre-deployment steps to here okay i'll just cut it and i'll paste it inside the before script section because these won't be like we won't be changing it uh much often so let's just keep it over there and whatever we have inside the script section let's just copy it and create one more script just cut it and at the project root directory i'll create a new file and i'll call it uh dot deploy underscore script dot sh i'll just start a new shell script pin bash and i'll paste it over here then i'll remove these iphones and like all the variables will be imported automatically you don't have to do anything about that just save this one as well and over here i'll just mention bash space dot deploy underscore script dot sh okay and i'll remove this extra lines okay this should be good now it's looking much cleaner it's you know easy to maintain now and let's just save it just click save so now i'm gonna stage these three files let's click on stage all okay i'll just put a comment over here added script a script and before script okay i'll just commit and i'll push the changes to the remote develop branch so once this is done as expected our build should be auto triggered let's go and check okay so our project was my first project this one and i'll go to cict pipelines and it triggered our pipeline so let's go and check what's going on over there so it's currently at the build job okay it's uh you know spinning up the docker container for our build job because we have used a shared runner for this so it's gonna take a few seconds and like earlier as well i'm not gonna pause this video because i want you to see everything in real time so let's just wait for few seconds and it will it will pop up some some results shortly just hit refresh okay so it triggered everything and let's read the logs so first it ran the before script which was you know echo compiling the code which is good and then it executed the bash script which is build script and it executed all the steps and it ran after script which says compile completed and everything is green everything looks good i don't see any error over here let's go back to the pipeline okay uh these two are still running in parallel we haven't made any changes to these two but let's just have a look so no error so far test okay and let's go back let's go to okay so this one would be okay it succeeded as well so it might not be refreshed by now so let's see okay so now this one is running on our private runner and it says job completed let's see what it did so it prepared the environment it initialized the git repository then it ran our prescript which was this up to here okay and everything is green everything is successful i don't see any error then it ran our bash deploy script which was our main script it did everything that was expected out of it it stopped a process because there was an already running process and then it started it and it was started successfully then it says job succeeded and this is done this is all good so now we can okay so this somehow because our pipe okay now it's showing as green this is how we could use after script before script and main script i hope this would be clear by now let's jump to another section let's now go through the grid flow that we'll be following in the upcoming section let's say we have three branches a developed branch a stage branch and a main a new branch gets created from the developed branch let's call it bug 101 and it gets merged into the develop branch with the help of once it is get reviewed and approved a build get triggered test cases can execute it and the deployment to the dev environments get triggered then a new merge request has been created to merge the same into the stage branch once it is reviewed and approved the deployment to stage environment get triggers then a new merge request gets created to merge the same into the main branch once it is approved and reviewed deployment to the production environment gets triggered and this completes our git all right so now that we have seen uh the git flow that we'll be using for this demo i hope you already know the importance of you know merge request or pull request and how effective and how important it is to use it in an organization which is following uh cicd practices using devops so let's uh go ahead and see it in real time okay so what we'll do is as part of that uh we'll create a new branch so we already have a develop branch and a main branch so we'll go ahead and create a new uh branch which is uh you know let's call it bug 101 so i'll click over here i'll click on new branch so bug 101 and this would be our local branch and it will push to the remote branch so i have created it and let's add some changes to this branch so that we could create a merge request so i'll just add after script over here and i'll call it echo bug 101 branch test that's it and i'll save it then i'll stage the changes i'll commit created the branch okay i'll commit the changes and then i'll push it to the remote branch okay it says publishing okay so now it wouldn't have triggered the build because we haven't specified any uh trigger condition with the name bug 101 okay so it will be triggered in the development environment only when the changes are merged into the develop branch so let's go over here and let's see if this is correct so yeah i don't see anything triggered as of now so let's go over here and let's see if the changes have been pushed so now i can see three branches over here main develop and bug 101 so let's go ahead and create a merge request so i'll click over here call it bug 101 but it's fine for now so just verify these things so it says new merge request from bug one to main so i don't want that i want bug 101 merged into develop branch so i'll click on change branches from here so this is my source branch over here and my target branch should be developed so i'll click over here should be good i'll compare branches and continue okay and now it change the title to create the branch it's fine by me and then i'll assign it to myself so that i could just approve it and it says delete search branch when the merge request is accepted because this was just a feature branch a short-lived feature rant that we have created so i want it to be gone once it has merged into the develop branch so let's keep it marked and i'll create create merge request so i have created the merge request and now because i'm the approver i could just merge it and i i can delete the source branch so let's do that as well i'll put a comment over here approved merge okay it says uh merged and source branch has been deleted as well and let's see if if it would have triggered anything as of now so i'll go to ci cd it triggered the build for us let's go over here okay so i'll pause the video for a few minutes over here and then we'll come back so i have waited for a couple of minutes and i can see the job is completed now um and all the stages have been passed successfully so i'll click on this one because this is where we made our changes and we'll go down and see so this was the echo statement that we have added bug 101 branch test so this is working fine uh we are all good with this so now let's uh go ahead and you know take it one step ahead and deploy this into a staging environment but for that we would need a staging environment okay so what i'll do is i'll clone the existing dev environment and so that it will have all the required prerequisites installed already and and we'll name it as a stage environment so what i'll do is i'll go over here i'll hit refresh so this is my ssh console for my dev environment okay so because it just deployed this project over here and if we go over here we see all the you know code as well as node module which was downloaded from the cache so everything is here so i'll just clean that up so that it it won't fail during the next deployment i'll delete these two okay i'll clear it and then i'll go back to my ec2 console and i'll create a server from this instance so i'll click on actions and images and i'll say launch more instances like this click over here okay so everything is same it's http port is open ssh board is open port 3000 is open this is good okay everything's good so i'll create one tag over here i'll just rename the stack so i don't see any option to edit maybe we could edit it later on once it is provisioned so it's fine click on launch i'll use this key pair acknowledged view instances okay so it's getting provision it will take a few seconds we'll do the prerequisites before that so meanwhile it's getting provision i'll just remove this one okay so this one is a new because it's in pending state so i'll just rename it i'll just call it stage or yeah stage is fine okay and i'll pick the public ip from here because we have to create a new variable in our gitlab pipeline so i'll go to settings ci cd i'll open this in a new tab variables so private ssh key we have used the same one so we don't have to do anything for this but i'll create a new variable let's call it stage server this would be the public ip the one that we have just copied and i'll click on add variable now we have to add this in our pipeline as well so let's go to the pipeline yeah it was there in the deployer script what we'll do is we'll override the variables now because we are using two variables because okay before that i'll just copy this and i'll create a new stage and i'll call it deploy stage job this stage is a deploy stage and i'll create one more stage over here as well deploy stage okay then i'll go down okay deploy stage my runner this uh before script is good except for the fact that it should have stage server and uh private search key is fine yeah this should be good but uh we need to make sure that in deploy script we are using uh ssh server as the state server not the dev server so i'll in this stage i'll create one variable over here and i'll call it server and i'll give this a name server so remember we had the state server set as the pipeline variable so this would override the value from here server okay and i'll create a similar one in the dev as well and i'll call it dev server okay so what we have done so far is we have created a new stage we have copied like cloned the existing dev job and created a new job for stage deployment we have created a new variable inside the job and called it stage servers the variable name is server and the value is state server so which is the pipeline variable and we are just overriding the value from here so if it would have been the dev deployment it would have overwritten this value okay dev server it would have replaced server with the value dev server and for the stage it will override the value of a server with state server and this value will be using in our deploy script which is over here so instead of dev server it would be just the server over here and over here as well okay this should be good and let's see if we need to make any changes the build script we have not used any variable inside this so this is fine so i'll just save this one and this one okay let's just first deploy this in our dev environment we'll do the same thing okay so before that let's just update this file as well and uh call it hello world 2.0 okay because we are just deploying the new version everything's good let's just uh we still have a branch bug of bug 101 because this is the local branch that we had and we deleted the remote branch only okay so i'll just keep this name as same let's say updated version 2.0 and edit stage okay before that i'll just stage the changes and then i'll commit to bug 101 this is done and then i'll push the changes okay this is done now let's go back to the server okay we have made the changes over here okay and we have pushed the changes so now let's go over it says invalid uh and valid yml so we might have an issue with a yaml let's go back and see okay so the error was probably due to i have missed uh like the keyword is not correct it should be variables and i'll also put that into double quotes so that it'll resolve it okay and the next one would be over here should be variables and double quotes okay this should be good now let's save this one state the changes and i'll call it test101 commit to bug 101 then i'll push the changes shouldn't uh throw any error now let's see [Music] okay we go to pipelines okay it did not throw any validation error so we should be good it did not trigger the pipeline because we did not merge it into the develop branch yet so which is good so let's just go and hit on merge request i'll create a new merge request and my source branch would be bug 101 my target branch would be develop okay this should be good and yeah let's hit on changes now i'll assign it to myself delete source branch create merge request yeah it triggered the pipeline wait for uh a couple of minutes and these both both these stages should have been triggered so i'll just wait for a couple of minutes and then we'll come back so i have waited for like few more minutes and i see all the stages have been completed now and so let's go and see if the latest version of the updated version of our app has been deployed to uh to the dev and state server so i'll go back to my ec2 console i'll click on dav server and i'll copy the public ipv4 dns and i'll append the port 3000 because this is where our forever process started our index.js so yep i can see hello world 2.0 this should be good and then we'll go back to the state server i'll click the public ip i'll copy it and then i'll paste it with both 3000 and this is available over here as well so we have completed one cycle we have updated our code we have deployed it into the bug 101 feature branch merged it into develop branch and created another merge request to you know to deploy it into the state server let's quickly go ahead and see the gitlab architecture that we'll be following in the upcoming section they have a gitlab server hosted with a sas offering or as an iis service on our cloud infrastructure let's say we have a developer who creates a merge request in a project repository to merge his code into the deployment branch approver gets notified about the merge request he reviews it and approves it once it is approved it triggers a build a good ci cd pipeline which has three stages build test and deploy which in turns triggers the first stage which is build the jobs get assigned to the gitlab runner which is the application that runs the job it then executes the job and publish the build artifacts to the artifacts as well as to s3 bucket this completes our first stage which is a build then it triggers the next stage which is test job gets assigned to the gitlab runner which then runs the test cases and publish the test results to the artifacts or whichever location we have specified this completes our second stage which is test then the next stage which is deploy gets triggered the job gets assigned to the gitlab runner it triggers the build it creates a ssh connection with the application server which is hosted on cloud or on-premises and then executes the command remotely by downloading artifacts and config from s3 bucket and revert back the status to with labrunner and this completes a deployment stage as well as the overall pipeline the user gets notified about the status of the bill s3 bucket can also be used to upload and download artifacts like the build artifacts along with the secret and confidential data so for that what we'll do is because as part of the ci cd process when we'll be creating a merge request to merge develop branch into the main branch so that it could trigger the stage deployment it would not have the artifacts available because only the stage this deployment stage will be triggered and there was no build stage triggered before this right sorry not this one this one so only this stage would be triggered and uh no build deployment or build creation stage would be triggered as part of that pipeline run so from where would this job get the artifacts from so it has to be stored somewhere at a central build repository or at a central location from where it could get the artifacts so what we'll do is as part of the build process after the bill's been completed let's say over here builds script so earlier we were just zipping it okay and just using it as an artifact so now what we'll also do is we'll just upload it to s3 bucket so i'll upload it to the same bucket i'll call it so i'll use the sudo aws s3 and my artifacts would be this okay and i'll copy the bucket uh url let's see where it is i'll copy the bucket url from here so i'll remove the file name so my source file and location and my destination so this command will upload the artifact to the s3 bucket okay and when we go to the gitlab ci so for uh dev deploy it's fine if we use if we are using the artifact but for the stage deployment it shouldn't be from that but it will be downloaded from the s3 bucket so i'll go and update the deploy script as well so instead of for scp-ing from here so let's remove this one okay and before this i'll add an aws command which is aws s3 copy from this location and i'll name the file over here which is my first project third sip and download it in the current directory which is uh home ubuntu and then over i'll do this and then okay this is the escape character so that it's like treats this as one single command even after the line break okay you could just remove this and keep it everything in one line okay whatever is fine by you i'll save this one so i've updated the deploy script i have updated the build script to have to have it uh upload the file to the bucket and i've updated the gitlab ci pipeline to download it no i haven't made any changes over here because i have made it in the these two scripts so this should be good so i'll stage these changes again i'll commit it to the bugs this one a bug 101 branch final build upload and download i'll commit the changes i'll push it to bug 101 and this time let's just merge it directly to the main branch to trigger the stage deployment so let's just go to the pipeline and let's just go to the merge request create a new merge request first the bill has to be created so select the source branch as book 101 and the target should be develop branch i'll compare it i'll assign it to myself create merge request and then i'll merge and delete source branch which is fine it should trigger the deployment now so i'll pause the video again so i have waited for some time and our build job is completed successfully so if we click on that we see our upload s3 command has been completed as well so it should have uploaded the artifacts to our s3 bucket but there was one issues with that i had to do uh some minor updates because uh you see we are running this gitlab runner as uh on our docker container so we cannot attach an s3 roll to the container so i had to add the variables to the ci cd settings so i had to add aws access key id default region and secret access key so that it can authenticate to s3 bucket and to upload the objects right so that was i did that behind the scenes okay now i can see it's been completed successfully let's see if it deployed latest code to dev server so i'll copy the public dns go to port 3000 and it uploaded the latest version let's see it up if it uploaded the artifacts as well so i'll go to this bucket again and i'll hit refresh i see my first project.zip over here as well so which is what we were expecting so as part of the final step of this demo so let's just create uh the last merge request to merge it into the main branch so that the stage deployment should be triggered so i'll click on new merge request select the source branch as develop target should be main click on compare at this time i would not be deleting the source branch create a merge request and i'll approve it it should have triggered the pipeline it says this pipeline so not this one okay so it says deployment stage and i guess it's been completed so all the steps it downloaded the file and it did the deployment i don't see any error let's see if it did the deployment successfully i'll copy the state server url from here open it in new tab for 3000 i'll see the hello world 3.0 so finally we were able to achieve you know ci cd using gitlab end to end and we covered a lot of topic that we could have covered in such a short time so i hope you would have learned something out of this lesson and you have enjoyed it so if you did i would request you to comment on the section below and provide your feedback let me know if you are facing any issues or any challenges while learning this section and if you are facing any errors as well just let me know i'll be more than happy to help you out keep watching our videos please subscribe the channel if you haven't done so thank you very much

Channel: Tech Tutorials with Piyush

Views: 4,360

Rating: 4.9024391 out of 5

Keywords: gitlab, gitlab ci, gitlab ci cd, tutorials, nodejs, gitlab runner, devops, devop, devops tooling, cicd, aws cicd, beginners, aws tutorial, aws, console aws, cloud amazon, aws certified, amazonaws, cloud, computing, aws training, certified aws, amazon aws s3, aws s3, aws ec2, amazon ec2, aws cli, cloud computing, aws tutorial for beginners, gitlab auto devops, s3 bucket, ci, gitlab tutorial, gitlab ci cd tutorial, gitlab ci cd pipeline tutorial, what is gitlab, amazon web services

Id: uYaMRn2meJQ

Length: 77min 48sec (4668 seconds)

Published: Wed Jul 07 2021