Multi-Stage YAML Pipelines | Azure DevOps |A Comprehensive Guide for Beginners | OmegaCodex

hi there welcome back to the channel I am sanal and today we are deep diving into the world of azure pipelines using yaml editor now if you have been following along in this particular video we took the first steps by building and deploying a.net core application using Azure Pipelines here we created a build pipeline using yaml editor as well as classic editor but then we use the classic release pipelines in Azure devops to deploy our build to an Azure app service but today we are taking it up a notch we are leaving the classic editor behind and fully immersing ourselves into the power and flexibility of yaml pipelines we'll be dissecting the key Concepts like stages jobs and stats and I'll show how to effortlessly manage both your build and release pipeline by wrangling it all into a yaml pipeline no more jumping back and forth diff between different editors with yaml you are in full control so let's Venture into the world of yaml pipelines don't forget to hit the like button and subscribe for more in-depth tutorials alright let's get started the latest way to build CI CD pipelines on Azure devops is with yaml files the older way was to use classic editor earlier we had created build pipelines and release pipelines using classic editor we also created build pipelines using the yaml file but we still use the release pipelines to deploy our application if you are new to these Concepts and are not very familiar with Azure device pipelines I would highly recommend you watch those videos you can find the link to those up here in this video we'll be focusing on creating the whole CI CD flow from building our application to deploying it on different environments using just the yaml pipeline editor now before we start creating pipelines let's understand more about the yaml Syntax for Azure pipelines yaml which stands for yaml end markup language is a human readable data serialization language it is commonly used for configuration files yaml uses indentation and simple punctuation marks to represent data hierarchies making it easy for both humans and machines to work with it it's a popular choice for defining configurations in tools like Azure devops Docker compose kubernetes manifest Etc in Azure devops yaml is the new way of defining pipelines we can Define our pipeline in a yaml file with rest of our application code this makes the pipeline also versioned along with our code it follows the same branching strategy you get validation on your changes through code reviews and pull requests and Branch build policies any change to the build process that could break the pipeline or result in an unexpected outcome can easily be identified because the changes is with rest of our code base now let's look at the components that make up the pipeline now the first component is the trigger a trigger is something that's set up to tell the pipeline when to run you can configure a pipeline to run upon a push to a repository at scheduled times or upon completion of another build a pipeline is then made up of one or more stages a stage is like a logical boundary in the pipeline it can be used to Mark the separation of concerns like build Dev QA or production when we Define multiple stages in a pipeline by default they run one after another we can specify the conditions for when to run each stage we can also set up dependencies between stages to make sure that a particular stage runs only when another stage is completed successfully for example deployment to death happens only once the build is completed successfully now every pipeline has at least one stage even if we don't Define it explicitly as we have also done it in the earlier video now it makes sense to create stages when you have different logical operations happening as part of the pipeline to have a clear separation it also helps in having better control like having set of approvals for jobs or separating a job that needs more running more time to run each stage contains one or more jobs a job represents a execution boundary for a set of steps all these steps run together on the same age jobs are most useful when you want to run a series of steps in different environments for example you might want to build two configurations maybe x86 and x64 in this case you can have one stage and two jobs one job for x86 and another for exit at 64. a step is the smallest building block of a pipeline a step can be a script or a predefined task which is nothing but a pre-packaged script that has been abstracted with a set of inputs an example of a task could be like publish the build artifact or deploy it to app service another important component is the artifact which is the collection of files or packages published by the Run artifacts are made available for subsequent tasks such as distribution or deployment I guess enough of talk let's jump into Azure devops and start creating a pipeline so I have a project in Azure devops as devops samples this is the project when I had used in my previous videos so here I have a dotnet core application which we will be building and deploying it to Azure as you can see I already have an Azure pipelines file here IML file which I had created in the earlier video and what it does is just build the source code and create the artifact in this case we had used the yaml file just to build the application but then we used the release pipelines to release it to Azure in this video we will be using only the yaml file to create the pipeline and it would do the complete process for us let's come to pipelines and click here to create a new Pipeline and it would ask where our code is so we'll select a repository as get so Azure device does offer you different options to use different types of repositories you can use the inbuilt Azure wrappers get or you can connect to bitcloud bitcoin get Cloud GitHub get bit Hub Enterprise server or any other git repositories so here we will be using the Repose get because Accord is here and when you select that it automatically shows the repository available under this project if you have multiple repositories it would show up all of them let's select this and let's just use a starter pipeline so that we have a fresh one and let's rename it and split stages and let's remove all other things and we'll start with trigger so I was mentioning earlier also that the pipeline starts with the triggers we have to mention what is the trigger so here we are just mentioning the trigger keyword and mentioning the branch name so whenever there is a code commit in the main branch this pipeline would be triggered so that's the trigger we are going to use next let's add some standard variables which we will be using in our pipeline so this is one way to set up variables in an Azure pipeline so variables are like we have variables in programming languages where you can reuse them throughout the code so similarly when we Define variables here we can reuse the variable names in different part of our yaml file if any changes are to be made to those values we just need to make it at one place and it gets applied to all other places so that's the benefit of using variables wherever we can after that the next step is to create the stages start with we'll use the stages keyword which lets us Define the stage under stages we would have to mention stage and let's give the name for the stage this call is built because we are as the first stage we are going to build our application and let's have a display name build application let's call it build application and then we have jobs I think there's a typo let's correct that so as I mentioned under a stage you can have one or more jobs so under the jobs keyword we would be defining our job in this case we only have one job that is to build our application you call it build and here we would need an agent so every job runs through an agent unless it's an agentless job so even for building our application will need an agent which would download the source code and build our application when you are using Azure pipelines we have got a convenient option to run our jobs using Microsoft hosted agents with Microsoft hosted agents maintenance and upgrades are taken care for us each time we run the pipeline we get a fresh virtual machine for each job in the pipeline please note that the virtual machine is discarded after that one job which means any change that the job makes in the virtual machine such as checking out the code will be unavailable for the next stop Azure pipelines provide a predefined agent pool named Azure pipelines with Microsoft hosted agents for many teams this is the simplest way to run jobs you can try it first and see if it works for your build or deployment if not you can use the second option which is self-hosted digit a self-hosted agent is an agent that we set up and manage on our own to run the jobs we can use self-hosted agents in Azure pipelines or Azure devops server formerly non-stfs self-hosted agents give us more control to install dependent software needed for our bills and deployments you can install the agents on Linux Macs or Mac OS or Windows machines you can also install the agent on Docker container how to set it up would be a topic for another video but if you are interested to know more about it you can check the reference links in the description box below so for defining the agent we want to use our agent pool we want to use we have to use the pool keyword and here we will be using VM image as Windows latest so what we are using here is Microsoft hosted agent with a VM image of Windows latest it offers different VM images like Windows Ubuntu and other flavors as well now once we have chosen our agent we have to specify the steps for our jobs so let's add the steps now steps are nothing but different tasks you have to do as part of your pipelines it could be either a script or a predefined task this stage we can you also use the task assistant provided with Azure devops which can help us in creating a pipeline so if you come on the right you can see there is an option called show assistant and here we can use this.net core task which would help us set up our pipeline so to start with let's say we do new Gateway store so the paths will have to specify what is the path to project and feeds to use let it keep what is by default and just click so it just comes sometimes little out of the indentation so let's intend it under steps as mentioning in in the yaml the indentation decides the nesting structure so since task is under steps so we have to keep it in properly intended so similarly you can add other tasks also to build your pipeline so in interest of time I'll just paste other steps required to build this.net core application I will just explain it so after restore we are building the solution once the solution is built we are running our unit tests and then we are publishing the code coverage result and then publishing our artifact as part of these two steps if you want to understand this in much more detail you can check out the earlier video where I have explained all these steps in much more detail the steps till this point is pretty much similar to what was done earlier the only difference being that we had we had to put it under a stage because this is a multi-stage pipeline and obviously the next step is to add one more stage which we can obviously do here we just minimize it and just have next stage which we can call to Dev my bad but there is a slight difference between the way we have done it using the classic editor and how we have to do it in this yaml file the reason being that each job runs in its own agent so when it finishes the tasks with that agent all the artifacts created with it may not be available in the next day especially when you are using a hosted agent for each time when the job is initiated a new VM is created for that job when it moves to the next stage the previous VM is deallocated so we wouldn't have we won't have those artifacts available then we are coming to this stage to fix that problem we will have to add one more task here which is called a publish task and we have to mention which item to publish so we had published our artifact of this path so we can just mention the same path here and we have to give a name for this ad so let's call it web app build so this is the name which would let us identify this artifact in the next stages so now when we come to this stage another keyword which we have to add is depends on because this stage will depend on the previous stage so let's say depends on build and we also want the condition to be as succeeded so it will only run if the previous stage had succeeded after adding these two steps we can go ahead and again add the jobs keyword to deploy our jobs now at this point since this is a deployment task Microsoft recommends using a different type of job which is called a deployment job in yaml pipelines Microsoft recommends that we put our deployment steps in the special type of job called deployment job a deployment job is the collection of steps that runs sequentially against an environment an environment government is a collection of resources where you deploy your application it can contain one or more virtual machines containers web apps or any service that's used to host the application being developed a pipeline might deploy the app to one or more environments after the build is completed and the tests are up a deployment job and a traditional job can exist in the same stage some of the benefits of deployment jobs are like we get the deployment history across pipelines down to the specific resource and the status of the deployments for auditing we can also Define how our application is to be rolled out Azure devops supports the Run once rolling and Canary strategies what are these probably a topic for another video type here you'll get the deployment option so earlier in the earlier stage we had used a job with the keyword job but a deployment we will be using deadline now with deployment job there is a slight change in in the steps which we have to Define we can have a display name and let's call it apply to Dev now it's it's no display name is an optional field it's good to mention it because when you run run the pipeline these are the names which gets displayed on the screen and it becomes easy to identify that okay which at which stage or what job is being performed and kind of helps us to identify the things better otherwise it uses a default name for the task which sometimes may not be easy to identify like for example you if you see here it's not post cli2 and probably this is the name which would get displayed so you don't know what is happening inside those Styles unless you go and drill down in the logs that which task is being performed so giving a display name makes it easy to interpret and understand it now again here we will need pool and a new keyword we will use here is environment now if you see here we have something called as environment option here as well in under the pipelines menu and if you open it you will see nothing is here so we can also create an environment here if you have virtual machines which you want to maintain here but otherwise we can do it through the pipeline as well so we'll come back to environments once we have run our Pipeline and then we'll see the magic so let's call it Dev because we are deploying it to Dev and now we will need to use variables for this pipeline the question might be that we already have variables towards top of the file then why we are using it here here we are using it because we could have different variable values depending on our environment right so we could have connection strings where example where the connection string for my Dev environment would be different and the one for my q a or production would be different so using variables here at this point will help us manage it now there are two approaches we can take at this point one is using the variables here you can click here and click on a new variable you can call it action string there and we can have a value if you want to keep this value a secret we can check this and it becomes hidden so once this is set as secret you will not and save it you will not be able to see the value so as it mentions here to reference we can just use this and and we can use those variables in our pipeline we have defined here and then we use it with a dollar value as mentioned here as dollar and then solution with the approach we will take now is a little different we will use libraries so what library it does it lets us create a variable group in most cases it would be that you would have multiple variables or multiple configuration settings which applies to any environment not just connection string but you would have some other settings which applies to your environment so in that case you can use this Library menu and create a variable group a library is a collection of build and release assets for Azure devops project assets defined in library can be used in multiple builds and release pipelines for the project the library tab can be accessed directly in Azure Pipelines here I have already created them but you can always use this plus symbol and create a variable group give it a name and then you can add your variables so if you come to libraries and just open this you can see I already have a connection string mentioned here for the dev environment and similarly I have one for QA and I can add more variables to these groups to use it in my pipeline so now let's come back to the pipeline and use the variable group so under variables I can mention group and I can specify the variable group name so once it is specified all the variables under this group will be available for this particular job the next one keyword after that is strategy so what kind of strategy we want to use for our deploy and we want to use run once because we are running once in our pipeline under run once we have to apply and after deploy we come to steps and from here on it's again similar like we did for the jobs in the above state so now if you remember in the previous stage at the end of the previous stage we published our artifact and that is what we want to use to deploy our application so to start with we have to download that so we'll use the download task and we'll specify current the current means in the current pipeline whatever was the artifacts from there we have we want to download we can also do it from a specific run another alternative to download is you can come to tasks and you can look for download pipeline artifacts it's same thing but then it comes as a task so you would have a similar structure like task and and the name of the task and then in the property so here like we did download current so the same thing is download artifacts produced by current run or a specific run and then we can choose the project so we can also choose it a different project from a different pipeline if you want for now we'll just stick to this one and we will call the artifact and green specify the artifactory this is the same name by which we have saved it and we'll using this we'll be using the same article now as the next task we will use a task called web app so we'll use Azure web app because that is what we want to deploy and it gives you options which we can select so I have already have a service connection defined in Azure devops which I have also explained in my earlier video so service Connection in simple words is nothing but an app registration on Azure which helps us manage this resources on Azure under a particular Resource Group so I have already set up this as your service connection here it's just a name which I have given you could name it anything and my app type is of Windows and I can choose the dev app I don't want to use a slot or an app service environment I'll keep it same and in the package I have is in a different location I'll just is this a pipeline workspace star start zip so under the pipeline workspace that's where my publish task would have published the artifact so from there under the subfolders pick the zip folder for deployment and also remember we have used variable groups and we want to set our connection strings or app settings so for that we'll use this application and configuration settings and in app settings we will Define our app settings so you can see the format here it's in this format hyphen and then the app settings name and the value for the next value again hyphen the name of the setting and the value of the setting if you have spaces or special characters in between it is advisable to put them in double quotes so that's the same thing I have done for this connection string default connection I have put the value under the double quote so with that done development deployment stage is complete so for the next stage it should be pretty much simple just copy paste for QA and we'll just rename it to a environment here we'll call it qae and here one thing we have to change is depends on depends on we want to depend we don't want to depend a Qi deployment and are built it should be dependent on depth right let's change the depends on to reply to left it will run only after it is deployed to test rest everything is fine and our app name for this was QA so if you you can always edit this task by just clicking on the settings link and it would show up here and see that it has selected QA sorry we have qae and rest everything is fine we have map settings and okay somehow our variables went off they've got replaced last time web so with that our pipeline is ready so let's just save it so ideally you should create a new branch and commit and do a PR depending on the process you have in your organization or your team but for now a Simplicity sake I am just doing it directly to Main Branch okay it says QA so let's just edit this too so let's come to pipelines and since we made a change our older pipeline started running for better understanding [Music] let me just rename this a pipeline is started running and we can see that we have three stages here so building application deployed to Dev and applied to QA so let's check this out it has started running and as you can see it is showing all the display names we have marked for each of the step and it is running one by one so let's give it a few seconds to run and Finish the build so now the build is complete but if we come to the summary page of the pipeline run you can see that this stage is waiting for permission so when we create a yaml pipeline and we are using variable groups the first time when it tries to access the variable to group it requires the permission to access that resource so we have to just click on it and we have to permit so it's asking for a permission for environment and as well as for the available groups so as it says granting permission here will permit the use of environment if for all waiting and future runs of this pipeline so we want to use the environment so let's give it that permission and also permit the access to variable group similarly we'll have to do the same action when it comes to qae stage and you can just expand and see that it is downloading the artifact and then it will deploy the application to Azure so let's give it a few seconds and it has completed the deployment to Dev and if we go to this we'll just copy this link and paste it here and you can see it is deployed and we have the connection string which we had mentioned for Dev so let's come to the QA stage which also would be pending for similar permissions so let's click on view and permit access to QA also permit access to variable group and with that it should go ahead and deploy to qae as well so let's give it a few seconds and deployment to qae also is complete let's just copy this link and open the qae portal and you can see that the connection string here is of the QA server which we had mentioned in the QI variable group so this way we have created a pipeline and a single pipeline file with multiple stages and we have built it and then deployed to two different environments now in an ideal scenario we wouldn't want the deployment to just happen as soon as it is built to be deployed to different environments we would also want some level of control right so that's where environments come in the environments which we used as part of our pipelines and if we come to this environment section we can see that it created those environments automatically under this environment space good always create beforehand just give a name for for an environment and resources we can keep it as none to add later and since app services are part of azure it's not as separate VM or acubernetes space so we don't have to select anything and we can use the environments or when we Define it in pipeline if it is not available it automatically creates it here and if you go to each of these environments it shows the deployment history has built now here in an ideal case one thus once the solution is built we ideally deploy at least one Urban element maybe a pre-diable everybody has different naming conventions but usually the higher environments require some level of approvals some monitoring to be done before it is deployed there or some some level of gates so for that for example for qae we can add Google and checks so we can have approvals we can allow deployments based on branches that are linked or maybe only during some business hours or different options so there are quite a few options which is available which we want we would want to do so as as an example we can try private approvals you just use my username as one of the approvers and here it is like all approvers must approve now if we come to pipelines and run this again let's see what happens so let's give it a few seconds to build and deploy to depth so now we have deployed till Dev again and let's come back to jobs and you can see that our deployment to QA is waiting and 0 out of one check has been passed so if you click on this it is you can see that one approval needs a review so it is asking for the approval of a reviewer whoever is marked in the approval list so since I have put my own username it will ask me for my approval to review and approve the stage so here I can give an optional comment like apply to okay and I can approve or I can also reject so it is asking for an approval for this environment and I can click approve and then it goes ahead and deploys it so this is something which was available as part of release pipelines in classic editor when you go to this release option at the release menu it's a similar functionality is available here also through environments where we can have approvals and checks with that we have created a full CI CD flow using yaml pipelines in Azure device this is probably the most common and basic flow which should be enough for many projects you can add more stages for additional environments and also enable approval gates for more control I have shared the code for the pipeline for your reference the link is available in the description box below let me know in the comment section your feedback questions and suggestions also let me know if you want me to cover any specific topic related to pipelines I'll be glad to help I'll keep covering more advanced concepts and pipelines for other scenarios in future videos so subscribe to the channel so that you don't miss out on those videos if you are interested in streamlining your development process using Branch policies in Azure device you can check out this video this can be useful for continuous delivery of your source code if implementing branching strategies for your repository is becoming a challenge for you check out this video to learn about git branching strategies and how Microsoft does it thanks for staying till the end see you in the next one till then adios [Music]

Channel: OmegaCodex

Views: 4,054

Keywords: omegacodex, education, software development, yaml, azure devops, azure artifacts, azure repos, yaml pipelines, azure devops tutorial for beginners, devops, azure devops pipeline, continuous integration, azure devops boards, visual studio team services, az400, continuous delivery, azure web app, azure devops artifacts, continuous deploiment, yaml tutorial, az 400, azure agent pool, yaml devops, azure release, continuous deployment (ci-cd) with azure devops, yaml pipeline

Id: QHVIEEjCF5M

Length: 32min 16sec (1936 seconds)

Published: Sat Sep 23 2023