Learn Complete GitLab CI/CD in 1 hour | GitLab CI CD Tutorial

gitlab cicd course in this course you will learn everything you need to know to start creating cicd pipelines and automating your application deployment using gitlab throughout the course we will be learning all important gitlab topics through hands-on experience so I've created practical examples where we set up cicd pipelines for nodejs application for Pyar application set up by old Learners and I try to explain all the topics through Haron with very less Theory and no ppts to bore you off so I hope you enjoy this course and last a good app after watching it one important request before we start the course if you are watching this course and you haven't subscribed to the channel yet please subscribe to the channel it takes a huge amount of effort and time to create videos like this so please subscribe most of you watch the videos but don't actually subscribe so please subscribe the channel and let's start with the course okay we are all set to start with our gitlab cicd course but if you're new to gitlab there are few prerequisites or some prior knowledge that you need to know in order to understand this course completely L the first thing is you should be familiar with Git and also know some important git commands that you can learn from this particular video along with this you also need to know what is cicd what are cicd pipelines what is continuous integration what is continuous deployment and you can learn all of that from this particular video I need to know this with that being said let's start with our gitlab cicd course let's start with the question what is gitlab and to answer this in simple way gitlab is a devops platform that provides you with so many features including Version Control System cicd security container registry and so much more that we are going to be learning in this particular gitlab course so in order to use all these features or to use gitlab you need to have a gitlab account and you can create a gitlab account using your email ID or using GitHub or bitbucket or Salesforce I I already have my gitlab account created using my GitHub so and I'm going to use that throughout the course so go ahead with creating your gitlab account and after you create an account you also need to verify it or validate it using your credit or debit card so that you can run your pipelines using gitlabs shared Runners and I'll be explaining you what shared Runners are in the course but go ahead with creating an account and also validating it with your credit or debit card don't worry gitlab will not charge or store your card it will only be used for validation so go ahead create an account with gitlab validate it to get started with creating your first cicd Pipeline on gitlab and in case you face issues in account creation or account validation feel free to let me know in the comment section but you can also check out the gitlabs documentation which are very very good to solve your issues or else you can also let me know in the comment section with that being said let's start with creating our first cicd Pipeline on gitlab once you have your account ready this is what you'll be able to see this is gitlab's homepage and in the project section you'll see all the repositories that you created for now it's very fresh so we don't see any projects then you can see groups or the projects that you part of then you have issues here which are the task assigned to you merge request to merge a particular branch and all the other things that we are going to be exploring throughout this course so let's start with creating our first cicd pipeline after we create our repository so I'll click on create a project option here in this I'll be selecting create a blank project and let's name our reper as cicd course so I'll say cicd course for deployment we don't have any deployment planed we'll say no deployment plan or you can also choose choose any options here so I'll say no deployment plan for now for reability Access let's make this repo as public and I also want to have read me so I'll say initialize repository with read me with these settings I'm going to click on create project and this will create a repository for me in gitlab that you can see here so we are inside cicd course repo and to set up cicd pipeline for this particular repo we need to create a file that will hold all the configuration inside it and I can do that by clicking on setup C ACD button here which can make things complex so to explain you from scratch I'll click on plus option here new file and create a file which will hold all the cicd configuration in gitlab if you want to create a pipeline which will hold all the cicd configuration it should be named as do gitlab hyphen c. yml this is the name you give for the file that will hold cicd configuration because gitlab will understand only if you give the name like this unless you change the name in the settings and you can also check the documentation so if you're new to gitlab cicd start by reviewing some of the commonly used terms the first is gitlab C.L file to use gitlab cicd you start with gitlab C.L file at the root of your project which contains the configuration for a cicd pipeline and this pipeline uses yaml so we are going to be naming our cicd file as. gitlab /ci yml you can also use different names but you need to change that in the settings for now you should always use dog lab hyphen CI only then your cicd pipeline will work only then gitlab will understand that this is a cicd pipeline file now inside this file you will have to create jobs job can be of different types depending on what you want to do let's create with a simple build job which is going to execute a sample script and to show you that in documentation again jobs pipeline configuration begins with jobs jobs are the most fundamental element of gitlab hyen C.L F and it is is used to execute a particular script so you can see this is job one which is executing this script this is job two which is executing this script similarly we are going to create a simple job which will execute a script that will show us a message on our pipeline so I'll say script and in this I'll Echo a message saying hello from gitlab cic CD course okay this is our cicd pipeline configuration now I'm going to commit it and let's see if our pipeline is going to succeed and show us this message or not so I'm going to scroll down let's add a message saying add cicd file commit changes and this will be committed to my main branch once I commit this you can also see gitlab will provide you with this validation here if the cicd pipeline or configuration is valid or not if there are any syntax issues or not once you do this you can go to build section and click on pip which will show you if the pipeline is running or not right now you can see the status is running it can be passed it can be pending it can be failed the status can be of different type right now it is running so it's running status you can also see who has initiated or ran this pipeline so it's me which is why it's my name here and right now we only have a single stage which is the test stage by default all the jobs will run in the test stage and you can see it says pass so let's click on this option here and it will bring you you inside this you can see there's a single job which is this job here and it's inside test stage the job name is build job now if I click on this job I'll get more information about this so this is where you can see the logs and let me try to explain you here so the log has few things the first thing is it is running on a gitlab runner a gitlabs provide you with few shared Runners that you can use and Runner is something Runner is a machine which executes the scripts that you define in your job going to be learning about Runners very soon so you just need to understand that Runners are a machine which execute your script that you define in different jobs next you can see a build has been created using Docker image which is of Ruby image so the image used to execute this particular script is Ruby next it is creating preparing an environment for our job it is getting the source from repository executing the script and lastly we have a message hello from gitlab cicd course this means our job has been succeeded and our pipeline has ran successfully you can also see more information about the job here so the job ran for 26 seconds it has finished in 1 minute uh there was no queue the timeout for this particular job is 1 hour by default this is the particular Runner that is used for this particular job this is the commit ID which ran the pipeline and you also see different stages right now it's only test and we also going to learn about stage and stages very soon so now we have successfully created a simple cicd pipeline and we have also check the locks which explains everything about Runner the image we have used and also different logs about our job now if you want to rerun this particular job you can just click on this option here which says retry and if I click on this retry option it will go ahead and run the job you can see the job is running again it's in running state right now and after it is done it will give me uh if it is passed or failed obviously it will pass because we haven't made any change or there's no error so this is how you create a simple cicd pipeline in gitlab going through the course I'm going to show you how you can change Runners how you can change this images how you can create complex cicd pipelines and lot more so let's start with creating a nodejs application and also setting up cicd pipeline for it so I'm in my vs code and I have a simple nodejs application that uses Express as a framework and runs on port 8080 to display a message this is my node application for cicd demo as this application uses Express we also have a dependency for for Express 4.6.1 in our package.json and now we are ready to create a cicd pipeline to build and deploy this on gitlab but before we go ahead and create a cicd pipeline let's try to manually deploy this on our local machine to understand all the steps that we will require to create a cicd pipeline now if you want to run this application locally you need to have two things in your machine first is nodejs so if you don't have it you can simply install it by running sud sudo AP install nodejs hyphen y now once I run this it says node GS is already installed for me but you need to install it or else it will not run along with node GS you also need to have npm installed npm is a package manager for node which installs all the required dependencies in our case the dependency is expressjs is Express so if I want to run or install the dependencies using npm I also need to have npm installed so I'll run Pudo AP install npm hyph Y and it says npm is already installed okay now let's go and run the command npm install this will install the dependencies inside our package.json which is required for our app to run if we don't have this dependency we will get an error let me show you that so if I run node app.js it gives me an error saying cannot find module Express so we need to install this module and we can do that by running npm install command once I run this it can see it starts installing something and now we have a package hyphen lock. Json and also a node modules folder which has the dependency inside it now if I run the same command which was node app.js our application is going to run and application is listenting at this port let's open this port and you can see this is my node application for cicd demo all right now we have an idea of how this application is running what things are required so let's go and create a cicd pipeline for this I'm going to create a new file the file name is going to be gitlab hyphen c. yml remember this is the file name you will be using to create a cicd pipeline in gitlab in this cicd file I will create two jobs the build job and deploy job build job is to install all the dependencies required to run this application and the deploy job to actually run this application so let's create the first job which is build job in this I'm going to say script inside the script I'm going to run the commands to install all the dependencies but before installing all the dependencies let's run the command AP update hyph Y and to install the dependencies we use npm install command but as gitlab uses Ruby image we are not sure if npm will be installed or not so I'm going to say AP install npm hyph y now that the APM now that the npm is installed let's run the command npm install which will install Express dependencies for this application okay this is our build job similarly I'm going to create a deploy job so let's say deploy in this I will run the commands to actually run the application so in the script I'll say AP update hyphen y first and this application runs using the command node app.js so before I install before I run the command node appjs I need to make sure node is installed or not and as this is a Ruby image gitlab uses Ruby image to create a build node will not be installed so I'll be running AP install nodejs hyphen y these are the same commands we ran when we were running this application locally now that the nodejs is installed I'm going to say nodejs sorry node app.js command now this is our pipeline now this is our pipeline configuration and we can push this to gitlab but if you see there are two jobs which are build and deploy and we don't know which one will be executed first and to find out let's push this on gitlab so to push this code from local machine to gitlab we first need to create a gitlab repository so I'm going to click on new project create blank project let's name this as nodejs application and I'm going to make it public initial uze the repository with read me and click on create project once I once I do that I also need to connect my repo with local machine so I'm going to copy this https link and run the command get remote set hyphen URL origin and the link once I do this I can confirm if my repo is connected using this command so it's connected and I'm ready to push it so right now if I run the command get Branch you'll see I have main branch and I don't want to push the main branch directly because it's not recommended to push on Main Branch you should be instead pushing on feature branch and then merge it this is a workflow you should be following when they working in production environments so I'll create a branch let's name it get Branch uh feature Branch now I'm going to check out to that Branch so I can start pushing to it and if I run the get Branch command again now you can see I'm using feature Branch all right let's push all this Cod I'll say get add Dot and if I do get status to confirm if I'm pushing everything or not okay now I'll say get comment hyphen M my app and then get push hyphen U origin and the name of the branch which is feature all right if you're confused about these commands please please check out the git and GitHub video it will be easy for you to understand because I have explained it more properly so now after putting my username and password it will be pushed and you can check it now if I refresh the page you will be able to see a new Branch here and you also get a message saying that you push a feature branch and right now you can create a m request along with this we have also pushed the gitlab hyen C.L file so the pipeline has been triggered now I'm going to click on this pipeline symbol and you will be able to see two jobs so there are two jobs here the build job and the deploy job as we have defined in our gitlab hyphen C file but we don't know which job is going to be executed first is it going to be the build job that will be completed first or is it going to be deploy job if the deploy job completed first it will give you an error because build is used to install dependencies that require to run the deploy and you can see the deploy has been failed so if I click on this and try to see what was the issue you can see it gave an error saying cannot find module Express this was the same error that we got because we didn't installed the npm dependencies using the npm install command so if this is the case how can we solve or how can we tell gitlab to execute the build job first and then the deploy job first so you can do this using something known as stages in gitlab so use stages to Define stages that contain group of jobs use stage in a job to configure the job to run in a specific stage or the order of items and stages Define the execution order of jobs so if you define the stage and tell the job tell the gitlab to execute the job in this particular stage first and then the later on it will go ahead in that way so if you read here jobs in the next stage run after the jobs from the previous stage complete successfully so we have to tell gitlab that execute the deploy later on after the build has been completed and we can do that using stages Let Me Show You by editing the pipeline we have here so I'm going to go to pipeline SE pipeline editor or I can click on code option here click on repository now click on the feature Branch so I will not push it or merge it to main unless my pipeline is passing in the feature Branch so I'll click on gitlab click on edit option here and then edit and pipeline editor so now that we know we can Define the order of the jobs using stages so let's put stages here inside this I'm going to create two stages one for build and one for deploy deploy so I'll say build underscore stage and then deploy underscore stage if you remember gitlab already has these as the default pipeline stages so if you don't Define any stage your job will be running in the test stage and I'll show you that in a second so now I've used two stages build stage and deploy stage so if you look here this gitlab CI is invalid because build chop chosen stage does not exist available stages are this this this so once you define stages you also need to tell which job belong to what stage so if I put build job in build stage it will be executed first because build stage is first and then there is deploy stage so I'll put stage and it belongs to the build stage so I'll say build underscore stage similarly for deploy I'm going to say deploy underscore stage and once I do that you will see this message will be gone so I'll say stage here and it belongs to deploy _ stage and now you you can see it's validating and now pipeline syntax is correct let's go ahead and commit the changes in the feature Branch commit message is fine so I'm going to click on Commit changes and this will initiate a new pipeline so let's click on Pipeline option here and you will see jobs running so let's click here and there are two jobs with two stages so build stage has a build job deploy stage has deploy job the build job is running whereas deploy is paused and will run only when the build has been completed so this is how you can use stages to define the execution order for the jobs if you put the first job in the first stage it will be executed first and then the later on so this is how you can Define and I hope now you have clear idea of what stages are in gitlab cicd so let's wait for this to be completed and then we can go ahead with see if there are any issues and try to troubleshoot it one thing that is one thing to notice here is every time a build is created it's always using image Ruby and we are creating an application that is nodejs so instead of Ruby we should be using nodejs so that we don't have to deliberately install npm or other options similarly if I'm creating an application in Python I should be using a python image so how can we change it and I'll show you that very soon well let's see if the build stage or build job has passed so okay so it says job succeeded build has been completed now if you look at the deploy deploy will start running now because build has been completed so let's wait for this to see if it completes or not all right so after waiting for some time we got an error saying saying uh job failed and why is that so job gave an error saying cannot find module Express so even though we defined the stages of build running before deploy it's still giving us the same error this is because two jobs do not share the information between each other so the first job was the build job which is going to install the dependencies for npm for Express and it is not sharing the node modules to deploy stage this is why it is giving an error saying that the dependencies are not installed so how can we solve this so to transfer information from one job to another or which can be dependencies which can be reports we use something known as artifacts in gitlab so artifacts can be used to transfer information or dependencies or reports from one job to another if you look at this example here we using artifacts for this job so in this example the job name PDF is calling this particular command and it's creating a file which is stored in the artifacts and under this name my cv.pdf so similar to this we are going to create artifact to store node modules which can be then transferred to the next stage which is the deploy stage and we can also download this artifact if we want to so let's go ahead and edit our pipeline so I'm going to go to repositories here and click on go to feature branch in the feature Branch click on gitlab hyphen C edit and let's edit this build stage to add another artifact which is going to which is going to pass on the Node modules that has all the dependencies required for this app to run so that we don't get the express error anymore so I'm going to say artifacts and you can see these are all the different options that you can have with artifacts when should the artifact run when should it expire and also it should be tracked or not so instead I'm going to remove all this as of now and just add a parameter which is going to be path so paths defined which file do you want to put in the artifact or which file do you want to pass on to the next stage or so if you look here the paths keyword determines which files to add to the job artifact all parts to the files are relative to the repository where the job was created so in this I want to add a file named as nodecore modules because this is the file that has all the information and the dependencies required for the application to run in this deploy stage along with this we also have a new file named as package hyphen lock. Json if you remember when we ran the command we got we also got this file for package hyphen lock. Json so these are the two files that I want to be passed to the deploy stage all right let's go ahead and push this so I'm going to say update artifacts in gitlab so okay once I commit the change it's going to trigger the pipeline and the pipeline is going to run again and once you have artifacts you can also download it you can also check what is inside that file right from your pipeline dashboard so if I show you here let's go to the pipeline and you have two stages again the build stage and the deploy stage let's click on build stage here and once this is going to be completed you will see an option here to check what artifacts are provided and if you can you can also browse them or download them in your local machine so you can see now the pipeline has been successfully completed and at this stage it says uploading artifacts for successful job jobs so it says uploading artifacts for successful job and you can also see job artifact section here you can keep them you can download them or you can also browse them let's click on browse option here and you will be able to see two files the node modules file and also the package.json file this has all the things necessary for the application to run so let's go back to the jobs and see if the deploy has been triggered or not so you can see the deploy job is now running and if everything is okay it should be able to pass or this job should also succeed you can see now the deploy logs are here and the application is running successfully it says application is listening at this point and this job will continue to run unless we stop it because unless we stop it so we have to make sure that this job is running on the background so that the next job should not be skipped uh because this is going to be in continuous loop and it will not stop unless we deliberately stop this pipeline so this is how you can use artifacts I hope now you have clear idea of how we use artifacts to transfer information from one job to another you can go ahead and check out you can go ahead and read about artifacts on the official documentation now let's see how we can troubleshoot this to run the job on the background rather than stuck in this particular Loop so I'm going to stop this pipeline Now by clicking on cancel and to execute this job in the background let's edit the pipeline again to add a script let let's edit let's edit the gitlab CI RL file again to run the job in the background so I'm going to go here in this I'm going to edit and I want the node app to be running in the background so I can do this thing which says run the app and then put the standard input output into a null which means it will be running in the background along with along with this we have also seen the image used used by gitlab for build is Ruby so instead you can Define what image do you want the job to be using and you can do that by image tag so you can Define what image do you want the particular job to be used and if you look at this documentation here you can define an image that used all jobs and list of services that you want to use during run time so you can also Define the image and also the services that being used so instead of using Ruby image let's define an image which should be node so that we don't have to install npm or we don't have to install nodejs for this particular job so I'll say image is going to be node similarly I'll also Define this here image is going to be node and so we don't need any of these now so I'll commment this out comment this out also comment this out similarly I don't want npm to be installed because it already be installed in the node app all right let's go ahead and push this so I'll say update image and click on Commit changes now after this changes we should be seeing that the image being used is node and also this is running in background without being stuck so you can see the pipeline syntax is correct and the pipeline has also been triggered so I'm going to click on this symbol here which is going to show us the two stages build stage and deploy stage let's click on it and instead of Ruby image we should be able to see that the image used is node now so this is how you can change the image in your cicd pipeline by defining the image tag so this is the image tag and you can Define what image to be used on instead of the default Ruby image so now the pipeline has been succeeded for the build stage let's see the deploy stage as well and you can see the deploy stage is is also running and it's using the image node so now we have successfully learned how you can change the image for your jobs one thing to notice one more thing to notice here we are using Runners here we haven't learned about Runners so let's explore what are Runners and you can see now the pipeline has been successfully passed so I can go ahead and merge this feature Branch into my main branch so now that the pipeline has been successfully passed I am ready to merge this feature Branch into my main branch so I'll go to repository and I'm going to click on create Mo request let's mge the feature Branch into the main branch so I can uh skip all this and delete the soul Branch after it is merged click on create merge request this merge I'm going to approve it and I'll the branch will be deleted once it is merged so I'll click on merge option here and once I do this all the code from the feature Branch will be go inside the main branch now so it says merge has been done and I'm going to click on Main here and you will be able to see all the code along with this it also triggers a pipeline because now the pipeline is triggered for the main branch so if you click here you will be able to see the pipeline which is now triggered for main branch you can see list of all pipelines uh here which has been executed till now so the previous one was on feature Branch now this one is on Main Branch and it's in build stage the deploy stage is waiting because we have us it because we have used stages in our cicd pipeline so now as you can see the pipeline for the main branch has also passed and this means we have successfully created a cicd pipeline which will automatically deploy our application for nodejs app so this is how you can create an application I hope you have now understood what are jobs what are St PES what are artifacts how to change the image in your job and lot more so moving on let's learn about runners in gitlab okay now let's discuss a very important topic in gitlab cicd which are Runners so in this section we are going to understand what are runners in gitlab cicd different types of Runners that gitlab provides us and also how you can set up your own runners in gitlab so if you look here in the code when we run our cicd pipeline it uses the runner that is provided by gitlab so gitlab provides you with shared Runners but you can also use your own runner for this particular job this is the runner we are using let me show you all the different Runners we have so I'll go to settings in cicd you will see an option which are here Runners so Runners are process that picks up and execute cicd jobs for gitlab so it's an application that is used to run the process or execute a service in a job so let's click on this which will give you bring you to documentation about Runners so gitlab Runner so gitlab Runner is an application that works with gitlab cicd to run jobs in the pipeline and there are two types of Runners gitlab SAS runners or shared Runners and self-managed Runners that you can set up your own now if I click on expand option here you will see the list of all the shared Runners that we have and right now there are available Runners 78 these are all the different Runners and you can also see the they have tags so this is Windows shared Runners this is also Windows shared Runner this is blue shared gitlab org blue shared gitlab org and lot more similarly if you want to set up your own Runner you can click on these or you can select this option and very soon I'll show you how you can set up a runner we will be setting up our own gitlab runner on Amazon ec2 so you can set this runner on Linux on Mac windows on Mac OS or Windows or Docker or kubernetes or AWS one of the reasons of using these Runners could be testing let's say you have a Windows application and you want to test it only on Windows gitlab Runner so you can set up your runner on Windows rather than using the Linux one so before we go ahead let's just understand the different types as I told you there are shared Runners and managed self-managed Runner so if you use gitlab you can run cicd jobs and SAS Runners hosted by gitlab these Runners are managed by gitlab and fully inte integrated with gitlab.com so by default these Runners are enabled for all project but you can disable them if you want to use your own and alternatively you can install gitlab Runner and register your own Runners on gitlab.com so if you look at this PPT here there are two types of Runners the shared Runner and sorry the shared Runner and specific Runners that you can install on your own uh server which can be kubernetes which can be Docker which can be ec2 instances or anything now how to use this once you install this Runner you can use it for a particular job by adding tags so I'll show you that in action very very soon but let's read about it more so gitlab Runner can be installed on any platform where you build go binaries because this gitlab Runner is created in goang so you can run this on any operating system like Linux Mac OS Windows FreeBSD cloud provider bare metal your workstation Docker anywhere and this gitlab Runner can test any programming language which can be net Java Python cphp and others so let's go ahead and set up our own gitlab runner for this I already have an AWS C2 instance created so I'm going to go to my console and connect to my instance with SSH now let's go ahead and set up our own runner in gitlab cicd and to do this I have an ec2 instance created in my AWS account and I'm going to click on on connect option to connect using SSH now to set up a gitlab runner you need to go to settings in cicd click on Runners and you need to click on these three dots now I'm going to click on show runner installation and registration instructions you can optionally go ahead and also follow the documentation if you want to so if you run if you run so if you search for gitlab Runner installation you will come across this documentation which will give you the steps to install install gitlab runner on your machine so depending on your architecture type if it is x86 amd64 or r64 you can choose to set up accordingly similarly the same settings are here as well so if you click on showrunner installations and registration instructions you have an option to select what your architecture is here you have options for Linux Linux Mac OS Windows Docker kubernetes AWS I'm using AWS Amazon uh so I can I choose to go with this but Amazon Linux I'm using ubben 2 so I'm using ubben 2 so this will also be fine so Linux amd64 386 arm or arm 64 if you don't know what is your architecture type you can simply run a command D package-- print hyen architecture once you run this you will get the architecture so this is EMD 64 and I'll be choosing this option here now I'll copy everything and create a script so let's go here type Nano before I do anything let's just run the command sudo AP update to update all the packages so I'll run this and it will update all my packages now I'm going to copy all these commands and put them in a script and run that script similarly you can also do the same uh if you look here these are the commands to set up on uben 2 so it's pretty much same I can use any of these I have tried setting up gitlab Runner using this as well as this this is bit easy and faster so I'll show you this option now so all the update has been done now I'm going to say Nano and script hyphen Runner Dosh in this I'll paste all the commands contrl srol X let's give it execute permission so CH mode hyph X and script hyphen runner. SH now I'm going to run sh script. runner. and this will start installing gitlab runner for me once this is done we can run the the next command which is to register the Randon using this URL and using this token so let's copy this and paste the command here all right now it's asking you the instance URL I want to use the same so I'll press enter for this then it's asking for registration token this is correct according to the command here so I'll press enter again I can add a description if I want to so let's go add a description Cloud champ or optionally you can keep the private IP if you want to next you can add tags for runners so tags are something which are required for a job to choose which Runners to run the job on if you see here in the tag section these are tags so tags are used to select a specific Runner CI tags are different from get tags and this is how you do it for this I'm going to add two tags here server and EC C2 next I can add a maintenance node if I want to I don't want to add that it's optional so I'll press enter next it's asking you for an Executor so there are different types of executor like Docker Docker Windows shell SSH Docker autoscaler all these options to better understand this let's go to this slide again and if you scroll down you'll see the executors here okay so these are all the common executors which are shell Docker machine Docker kubernetes if you want to run the commands as into a terminal which is a bash or sh shell you will be using shell some same for command prompt or power shell as well and if you want to run the commands uh in a Docker machine you'll be using this option similarly if you want to execute inside a Docker image you will be using this one the same for kubernetes and there are other options like virtual box parallels SSH we are going to be running this in Shell so terminal I'll put shell here and press enter now the configuration has been uh saved in the this particular file and it says Runner registered successfully to confirm I'm going to run the command sud sudo gitlab hyphen Runner and Status will tell me if it is running or not so it says service is running and if I come back to my gitl repo and refresh this page let's go to Runners you will see a runner has been created and it has a tag as Cloud champ it has the name as Cloud champ and these are the two tags we have given for our runner now let's go ahead and create a cicd pipeline which will now use our particular Runner to execute the jobs to test out the gitlab runner that we have installed on our ec2 instance I have a flask application created here so this is a python code which uses flask module and also OS module to create a simple application that showcase a message Dash is a gitlab pro where name is a variable and by default the value for the variable is John unless we pass another name so it will show us John is a gitlab pro and we are going to use this application setup cicd pipeline which will be running on our own Runner that we have set up on ec2 instance so before I go ahead and Commit This let me show you how you will be running this python application in your local machine in case you don't have any idea on how you can deploy or run a python file so I have a file already created if I show you this is my file which has the same code and this file uses flask module also OS module so if you don't have flask install you will get an error to install flask you just need to run the PIP three install flask command and you can see uh flask is already installed for me so it didn't took time to run the script now I'm going to run the command three and app.py this is the command to run python applications so once I run this you can see a python app is is running and if I open this up you will see John is a gitlab pro the message that we have defined here so I'm going to go ahead and commit this uh in my repo so add a new file message is fine I'm going to commit it in the main branch click on Commit changes along with this I'm also going to create a Docker file for this particular application and then set up cicd pipeline for it so I'll go here uh in the code option let's click on repositories and I'm going to create a new file for our Docker file so let's click on new file here and name it Docker file so Docker so I'm using the image python 3.8.0 the work directory set to SL app I'm adding all the code from my current repo into SL apppp installing the flask uh and then providing envirment variable as Mark and then running the command to start the python or to run this python app.py once I do this I'm going to click on uh let's change the message as ADD Docker file and then click on Commit changes option here now we have the application and its Docker file ready we can go ahead with creating a CCD pipeline that runs on our own gitlab Runner so now to create a CD pipeline for this application we are going to have so now to create a CV pipeline for this application we will now create a file as we all know we need to create a file with the name gitlab hyci yaml which will hold all the configurations for a cicd pipeline in this pipeline I want to have two jobs the build job and the deploy job so I'll say build job first in the build job I will run the command to create the docker image from the docker file that we created right now along with the docker image I also want to see the version for the docker so I'll run the command Docker version which will display the version in the logs next I'm going to run Docker build hyphen T to tag the image or to give the name and I'll say python app or PI app and the location for the docker file is current so I'll say dot here this part or this job is going to now create an image and also display the version now if I want to run this using my own gitl Runner I also need to add the tags so I'll say tags here and for the tags we need to use the same tags that we have used while configuring our gitlab runner so I'll open this in a new tag just to check as you can see the tags are ec2 and server so I'll be using the same such that gitlab will run this job on that particular Runner which we have set up on our ac2 instance so I'll use a tag which is ec2 and also server all right so this is our our build stage sorry build job we also need to create a stage so that we can Define that the build should be running first and then the deploy so I create stages and in this I'll create two stages one for build so I'll say build uncore stage then deploy underscore stage and I also need to Define this here so I'll say uh stage is going to be buildor stage all right similarly I'm going to create a new job uh which is deploy job so let's create it and name is name for the job is deploy the stage belongs to deploy stage and in the script I'm going to run the command to run the container which is Docker run command so I'll say Docker run hyphen D to run this in detach mode in the background hyphen iph name for the container I'll say uh python container container hyphen P to add the port so I want this to run on Port 80 so I'll say Port 80 mapping to 8080 and then the name of the application uh name of the image which is pi app so I'll say PI app here all right along with this I also need to add the tag similar to what I added here because we also want this job to be running on the runner that we have associated rc2 with once I do this I am going to commit this file so I'll say add cicd file and I'm going to click on Commit changes option now this will start triggering you can see the cicd configuration is valid and if I go to build click on pipelines you will see a pipeline has been initiated it failed though let's click on this to see why it failed actually so I'm going to click on build option here and and it failed because uh let's click on this link okay we have we have got an information here which says to trouble shoot this error check this thing and a script section comment it out and restart the pipeline okay let's go and check this thing so I'll copy go to instance uh do p sudo Nano and paste this once I do this okay this is the thing and they want you to comment out so I'll go ahead and comment this out also this line and this line okay let's go and check if our runner is running first so it's running and I'm going to rerun the pipeline so do this which will execute the job uh again and you can see it's running but it failed again saying Docker command not found all obviously it will not run because Docker is not installed here so I need to install Docker before I actually go ahead and run the pipeline because it is running in this particular Runner which does not have Docker install if I show you and I try to run Docker version command it will give me the same output which is Docker command is not found so let's go and install so let me install the docker first and then I'll rerun the pipeline so now Docker is installed and I can verify this by running Docker D- version command all right and you can also see there are no images right now in my machine once I run this it should execute the pipeline and also show me a image uh but we don't have gitlab runner in Docker group so it might give you an error so right now you can see the runner we are using is cloud champ which means we have successfully registered our runner to be used by this particular job and it has failed with the saying permission denied while trying to connect to Docker demon this is because we don't have gitlab runner in Docker group so we can do that by running Docker sorry Pudo user mode hyphen a capital G Docker and then the user which is gitlab hyphen Runner once I do this let's restart the docker service so I'll say sudo service Docker restart all right with that done let's try to rerun the pipeline again and see if we can get the job succeeded this time or not all right so now it has started creating uh the image and you can see there are six layers to this image which is which means now we have successfully able to uh run the job using our Cloud champ Runner so I hope now you have an idea of how you can use your own custom runners or how you can use your own Runners instead of using shared Runners there are many benefits of using your own Runners uh as I said testing is one of them second could be scaling maybe you have a big application that requires huge amount of compute uh and then your shared Runner is not capable to do it then you can have your own Runner which can be a kubernetes cluster or it can be a large ec2 instance now let's see the deploy stage and if you look at the deploy stage deploy has passed and we have job success seed message here and you can also get you can also see the Sha message which is for Docker run if I go here in my ec2 instance and try to run Docker images command you can see there's an applic there's an image for python app that we created which was 59 seconds ago and also python because we used this in Docker file if I do Docker PS I will be able to see a container running which has the same ID as we see in the uh Docker run in the in the logs here we can also test it out by just copying out this public IP and pasting it in the browser you can see Mark is a gitlab pro so this is how you can use your own custom Runners I hope you have now an idea of what is a runner how you can set up your own Runners and different types of Runners that gitlab has to give to and to use any of these Runners we know that we need to use tags so now you I hope you have an idea of what is Runners and if you have any questions feel free to let me know in the comment section moving on let's look at variables in cicd in cicd so what are what are variables in gitlab cicd if you're familiar with any programming language Let It Be python JavaScript or anything you might already know about what variables are variables variables are used to store values that you don't want to repeat again and again and and the value can be string can be numbers can be bully and anything similar to that gitlab also has variables there are different types of variables gitlab has you can Define your own variables or you can also use predefined variables that gitlab has for example you can check the list here on this particular link gitlab provides you with all this predefined variables that you can use throughout your cicd pipeline so if you want to get the author you can get using cicd commit author if you want to get the branch uh on where you're running the pipeline you can use the CI commit branch similarly message reference name all these different things let's try to St it let's try to run this and check if it actually works or not and to do that I'm going to quickly go ahead and create repo for variable demo all right in this I'm going to create a cicd file so I'll save new file name of the file is gitlab Hy C.L let's create a job named as demo in this inside this I'm going to run few scripts or commands the first is going to be echo echo the job name so if you want to get the job name in your uh log files you can see the predefined variable for it so this is the predefined variable cior jobor name this will give you the name of the job so once I put this let's also try to check more of these so let's try to get the uh job status job State Job timeout job token along with this you can also see M request page domains pipeline ID and lot more so let's get the runner so let's get the gitlab user ID here who is the user that is running this pipeline so I'm going to paste this let's save say paste this and click on Commit option below once I commit this should initiate a pipeline and the pipeline should show us the name of the job and also the username in the logs so I'll go here click on build click on pipelines and as you can see a pipeline is in running state right now let's click on it and click on demo the name of the job you can see the pipeline is in running state right now the logs are coming up so these are predefined variables in skit laab that you can use for your that you can use for your cicd pipelines you can see the logs are logs came up and this is the name of the job which was demo uh as you can see here this is the name of the job this is the gitlab user ID if I show you this is I can show you this is my user id 9131 3167 so these are predefined variables in gitlab that you can use and get the information right from the gitlab rather than you remembering it or putting it manually so this is one type of variable another is custom variable that you can add yourself let me also showcase you how you can use custom variables in gitlab cicd to create a custom variable you need to define the variables in the gitlab C.L file for example let me show you here so I'm going to remove all this here and to Define a variable in gitlab C you need to use the keyword which is variables here and variables are key value pairs so I'll say name name equals to Value all right similarly you can also say message equals to free and then I'm going to create demo job here to display the variables and use it in our cicd job so I'll say script and in this script I want to Echo a message that will be using this particular uh variabl so I'll say Echo hey message subscribe to name with that now it says pipeline syntax is correct so we can go ahead and click on Commit changes so defining a variable is very similar to what you do in any programming language you have a key value pair we are going to use the variable using a dollar symbol here so now the pipeline is running and if I click on it you will hopefully see a message that has the values of these variables in the result so let's wait for this to be executed all right you can see it has been executed and it says hey please subscribe to Cloud Cham so yeah this is how you can use custom variables uh similar to this if you look at the documentation variables can be of two types Global variables if you're using a custom variable uh variables are of two types you can Define them globally or through in a job so this is a global variable that is defined outside a job whereas this is a job variable that is defined inside a job and it will override whatever is defined in a global one another type of variable in gitlab C is secret variables that you can find here in settings cicd and variable section to demonstrate this I'm going to use the application Docker application to demonstrate this I'm going to use the python application that we created Docker file for and I'll be adding my Docker username and Docker password in this variable section and push the image to dockerhub so you can see these variables are used to store information like password secret keys that you can use in javascripts so if anytime you want to use AWS access keys or your username or your passwords that you don't want to expose in the script you can put them here in cicd variables so let's create a variable for our Docker Hub username so I'll name this as Docker hubor username and my username is Nasi 101 similar to this I'm going to click on ADD variable but before I add it let me explain you this so protect variables are variables that you can be used that you can use with protected Branch or protected tags in gitlab then you have M variables so if anytime you don't want to expose your value in the logs you can use mask variable option here so I'm going to uncheck this because we don't have the main branch as predicted and I want this to run so I'm going to uncheck this extended variable which means this is the variable that whatever starts with dollar so I'm going to click on ADD variable option here similar to this I'm also going to add my dockerhub uh token so let's go to dockerhub here and create a token for this so I'm going to click on account settings click on security and create a token for myself so let's say gitlab _ temp and click on generate option and this is the token that I want to use so I'm going to copy this and go back to my variable section click on ADD variable I'll say Docker uncore token and put the value here once I put the value now I can run the command I I can run the command and use the variables instead of using the username and password let's try to login using Docker login and also Docker push to push the image here in our Docker Hub so right now if I show you I don't have any uh image with the name PI app here and after I push it through my cicd I should be able to see it so let's go here in our code repository and come back to our gitlab hyphen c. ampy and this I'm going to edit it with edit in pipeline editor first before after I run this I'm going to run the command Docker login hyphen U and then the name of the variable which is Docker uncore username hyph p and then Docker uncore token all right if you look at this uh command here Docker run hyphen D hyph hyphen name python container hyphen P this is using a name and if I try to rerun it it will give me an error saying that the name already exist so one way is to remove the name and let Docker give you a random name or else we can add a command to delete this image this container before running it again so I'll say Docker RM python container before I stop it I need to also before I remove it I need to stop it first so I'll say Docker stop python container container and this all right so once I run this command it should me it should give me a message saying Docker login succeeded if the username and password is correct once I do this I'm also going to tag the image with the name uh Docker tag Docker T PI app the name of the here to the name I'm going to tag this image PI app with the name Docker uncore username SLP app so that I can push it and I'll say Docker push and then copying this thing here all right now this should work um and if I check pipeline syntax is correct I'm going to click on Commit changes option here and let's see if we get an error or this gives us the response so we're using secret variables in our pipeline to for our dockerhub username and also for our dockerhub to token so let's go here and you can see the build stage has been passed we are not using any variables there so no need to worry about that let's go to deploy stage and see if the deploy uh is showing using the variable or not so if you look at this it says cannot perform an interactive login from a non TTI device after checking I understood that I have used the wrong uh name so I have used dockerhub username and in here I'm using Docker username just so let's to make it easy uh just name it as username and password other than adding Docker or dockerhub you can see it's dockerhub username but in the cicd file I used that's I'll just name it username for now and click on edit variable similarly I'll do the same for Docker token as well instead of Docker uncore token I'll just make a token all right let's click on edit variable and let's go back to our pipeline also change it there in the pipeline so go to repository go to skit laab Hy C edited pipeline editor and instead of Docker and dockerhub I'll just make get username and password username and token so username and token similarly here I'll use username and username here okay let's change it and click on Commit change which will rerun our Pipeline and hopefully it should pass and we should be able to see the pipeline successfully passed so this is secret variable anytime you want to use any of your passwords or your access keys or your tokens make sure you use secret variables rather than hardcoding them in your cicd pipeline once that being said let's go and check our deploy job here so I'm going to go to deploy job you can see it's running the command to stop the container and remove it and it says job succeeded you can see login has also been succeeded and it has also pushed it so let's go to our Docker Hub and refresh we should be able to see an image being pushed with the name as py app so you can see image is pushed with the name as py app similar to secret variables another kind of things or another important thing that you need to know are keywords in gitlab similar to variables in gitlab you also need to know some of the important keywords in gitlab that will help you create create good create better cicd pipelines so these are all the different keywords which are default include stages along with this you have spec after script allow failures artifacts and lo lot more so to give you some context we use after script and before script for things that you want to do before the deployment so before script is used to define a command that should be run before each job including deploy jobs but after the restoration of any artifacts similarly after script is used to define the command that will be running after each job including the failed ones so you can use this keywords to run the things before or after the script along with this you also have another one which is only and accept so if you define that they only run on this particular Branch or only run on Main Branch or run it everywhere except the main branch these are some of the keywords that will help you enhance your cicd pipeline and you can check out all of them in this particular dashboard or on this particular documentation similar to this there are a few more things that I want to explain you before we complete this course now if you look here gitlabs provide you with many other things if you click on analyze option here you can see see cicd analytics of how much time your pipeline has been failed what is the success ratio uh and you can also change it depending on the duration so for last week last month last year and you can also see the graph here along with this lab gives you an option to check if your cicd configuration is correct or not right inside the uh editor but you also have an option named as CI lint which will Pro named as CI lint here to check your syntax if everything is correct regarding your get laab CI if there are any issues CI gitlab CI will let you know to give you an example let's go and copy one of the gitlab CI yaml files here so I'll copy this and paste it in the CI link option here and click on validate and this will give you if syntax is correct or not and all the other things let's let's try to make a change and try to uh change something so instead of this I'll click on validate option and it give you an error s syntax is incorrect CI configuration validated so these are some of the things apart from this apart from this there are few more options that you need to know in case you're using gitlab in your production environment so if you want to if you want to deploy your applications on different environments let's say staging or production or development you can use environment section here and you can create environments either through UI or you can also Define it right in your cicd pipeline so you can uh use product environment and deploy it on a particular ec2 or particular server and then use a staging which will be deployed on another server defining the external URL here along with this there are a few more things like pipeline scheduling and timeouts so if you by default whenever you run a pipeline it will run for 1 hour it has a 1 hour timeout so let me show you by default whenever you run a pipeline it will be running unless you stop it or unless one hour timeout has been completed so if you look here for this particular job the timeout has been set to one hour by default but this can be changed if you want to change it in the cicd section so you can always change the timeout here instead of 1 hour you can make it 10 hour or anything you want if your jobs are taking longer and it is failing due to timeout issue next you can also schedule your pipelines based on cron jobs if you want the pipeline to be running every uh every 12:00 in the afternoon or anything you like so you can just click on Pipeline schedule click on create a pipeline schedule option here and Define the crown job uh so this way you can set up a pipeline so let's say my important pipeline that I want to run every day at 5:00 and then I can select time zone so I'm in India so I'll say IST here or India let's see this one even though this is not the Indian time and I can choose to put a variable if my cicd file has a variable and click on create pipeline schedule and this will create a pipeline schedule and also tell me when will the next cicd pipeline run so it will run in next 14 hours so this way you can set up a pipeline schedule and we have seen a lot we have seen we have pretty pretty much covered everything about gitlab if you have any questions about anything in this please let me know also let me know if you want a full-fledged project similar to what we did in Netflix or using genkins we can also create something like that using gitlab if you have that request let me know in the comment section I hope this video was helpful finally this is the end of our gitlab cicd course I hope this course was informative if you have any questions any doubt do let me know in the comment section also let me know if you want a project where we set up cicd pipeline using gitlab similar to what we did for Netflix project I hope this video was informative if you found this video helpful please like this video And subscribe to Cloud champ thank you and have a good day

Channel: Cloud Champ

Views: 13,673

Keywords: gitlab ci cd tutorial, gitlab cicd course, gitlab ci/cd, gitlab tutorial, gitlab cicd, gitlab course, gitlab ci/cd tutorial, gitlab ci cd, gitlab cicd for beginners, gitlab tutorial for beginners, cicd for nodejs app, gitlab runner, gitlab complete course, gitlab, gitlab ci cd project, gitlab ci, gitlab cicd from zero to hero, gitlab cicd pipeline, gitlab ci cd pipeline tutorial, gitlab ci cd docker, gitlab pipeline course, what is gitlab, gitlab devops tutorial, cloudchamp

Id: JWXVijJfnHc

Length: 70min 14sec (4214 seconds)

Published: Sat Nov 11 2023