Azure DevOps - YAML pipelines with multiple stages

ml pipeline that pipeline will be available for you as a source code file in your Repository we have seen that now classic pipelines are very good they are running fine you can do many modifications many things why we need to move to yaml Pipelines what is the importance of ml pipelines and what are the benefits of ml Pipelines the family is that much good why people are still using classic pipeline So when you say ml is very good definitely see whenever a software or any application if it is completely fine then no one will use it okay because no company will make a completely fine product by knowing even with their knowledge wantedly they will make some tissues are some problems in the software application with the issues and with the problems only they will release the product into Market these are our marketing strategies if I give you one thing which is completely working fine then you never come to me so that's why I will give you something which you will get problems after one month or six months or one year so that you will come back to me so this way classic pipelines has some limitations people are not happy even though they are easy to create they are easy to use but they have some limitations some functionalities are not available so classic pipelines cannot be shared and cannot be recreated so damels are easy and the MLS can be saved as a file MLS can be saved as a file in your source code repository so that if tomorrow someone takes your source code along with your source code they will get your pipeline as well they just need to go to Azure pipelines create a pipeline and import the existing ml file now let us see our ml fit so this is our simple yaml file we have a trigger we have a pool we have a job the job name and the step so when I go to yaml Pipelines okay so here I am creating a new pipeline but when I go to religious so here if you see new release pipeline so do we have any option for yaml here nothing they have directly given a template to select so if I select the template it will create me a classic pipeline only it will create me a classic pipeline only so let me choose some some random bilateral this is the release pipeline okay so is there any option to create a release pipeline in the mL no there is no option to create a separate release pipeline in the ml then how can I create a release pipeline in yaml that's the point that comes the one more advantage of ml Pipelines so ml Pipelines are very useful when you wanted to create both the build and release in single pipeline why because Camille will be used to have multiple stages okay here stages means not the release pipeline stages in the ml pipeline stages means first stage will be build job second stage will be tested job third stage will be deployment job now I will show you a sample ml pipeline wait a minute foreign so here I have trigger is common for all the pipelines then pull then variables then if you see here stages so this stage is built so which means this entire stage at this entire stage will contain the build steps whatever we do as part of build pipeline this will contain all the build steps the next stage is deploy so this stage will contain all the steps that you do in release pipeline so here we are creating single pipeline for both build and release so yaml supports integrating both pipelines into one single pipeline so there will be there won't be two separate pipelines like building pipeline release pipeline if you are creating the ml pipeline which means you can first create build a pipeline if you want to release the same you need to convert the pipeline into stages first stage will be build stage second stage will be deploy stage so in the build stage you will Define the jobs we already know the syntax see here so the jobs inside the jobs there will be some steps and the steps will have some tasks same so here I have jobs inside the jobs I have some steps those steps are nothing but tasks first is new get installed then you get restore the visual studio build in Visual Studio test then publish the bill artifacts then publish the build artifacts now let me show you a classic pipeline where you can see all these steps there are four steps right thing five yeah see one task second third fourth fifth so there are five tasks in this pipeline you can see those five tasks see one two three four five so nuget build test publish foreign next one is build see this is build okay then build next one is test C test then test after that publish after that publish where are you publishing fill the artifact stage in directory you can see here see build artifact staging directory so all these classic steps are converted into yaml so here I have all the steps grouped into one stage that stage name is called build then deploy so here I don't have to apply steps I need to go to religious so I have a release pipeline where I wanted to apply to azure yes now here if you see so what I am doing in release pipeline there is single task okay there is single task with a job there is single job single task this is the task this is the job so in job you see we have an option artifact download okay artifact download so first you need to download the artifact then you will deploy here you see so this is my deploy stage First Step see here you have tab step steps here I have steps first step is download artifacts then apply to Azure web app they apply to Azure web app you can see everything so connection type Azure RM connection name Azure connection here you can see so connection name Azure connection right then the app service type web app and windows okay so here no app service type but the app name is there my app 929 see here I have given different name rajes website 929 earlier this is my app nine to nine so that is a web app that is a web app so packet system dot artery Factory which has zip file name you see system Dot working directory which has zip file which has different so this artifactory directory comes here from here why because you have downloaded the artifact to this folder so when I am deploying I need to use this folder only in this case that is different folder see here default working directory this one this is default working directory so I am taking the things from default working directory but here I am downloading it to different folder this is all about name only I can give any name this way you will have both release and build a pipeline into one single pipeline when you are creating pipelines in the ml it is another best example I mean another benefit another Advantage with Yami okay so when you are creating pipelines with ML there won't be two different pipelines for build and release both the steps whatever you are performing as part of build and release both will be integrated into one single pipeline that is the best part even the single pipeline also even the single pipeline also it will be committed to your repository see here I am into Master Branch sample website I'm into Master band sample website so if I go to this Repository here I have a sample website and master Branch see I have a pipeline Azure pipelines Dot yaml yes right Azure pipelines Dot yaml even this multi-stage EML pipeline also will be committed as a file to your Repository if tomorrow someone takes this repository or close this repository or even they can download this repository download as a zip they will get this pipeline they will get this by reply so they don't need to worry hey man I got the entire code but I don't know how to build it no need to worry you have the pipeline already there as committed as ml file inverse repository itself just to use the ml file and create a pipeline so the pipeline will build for you and the pipeline will deploy for you it will apply also it's only that observe the pipeline so what is the app name so my app 929 and what is the connection name Azure connection make sure that you create this app in Azure and create a service connection called Azure connection and create a service connection called Azure service connection so there are multiple tasks and multiple options can I modify these things because these all are written in a code but till today we are habituated for classic Pipelines how we expect we expect a task should be opened and add the values modify the values but here how can I add the values how can I modify the values so if you clearly observe our every task above the task title we have some button settings so we have a button called the settings if you click on the settings it will this will open our classic pipeline task editor page see here so this is azure RM web app deployment task if I click on the settings so it opened the settings page now being an engineer who worked mostly on classic pipeline now this is easy for you to modify if you are leader manager asks you to write all these steps in this code itself you may be confused oh God I don't know ml properly how can I write this code so nothing to worry if tomorrow your manager or your lead want to add one more task so here so here they wanted to add one more task so go to the line keep your cursor on the on itself in your emulator page there is a small button called show assistant this is so assistant is nothing but added tasks in our classic pipeline this is show assistant is nothing but let me show you I think this is look foreign so you got all the tasks the same way in your email also if you want to add additional tasks you don't know how to write the code just to keep the cursor of the required Place see if I keep cursor here it's okay now if I go for show assistant say I got all the tasks these tasks are nothing but these tasks but the only difference between classic and the ml is you will not have this categorization you will not have this categorization they are openly displayed no categorization whether it is build task why there is no categorization just now I told you the ml pipeline is not two separate pipelines they won't be build pipeline there won't be release pipeline the ml is both combined if you want to build there will be only one pipeline if you want to build and release there will be only one pipeline even if you want to build a test release deploy or install package anything everything will be single pipeline that way there is no categorization in yaml tasks RR comes under one single pipeline only but here there is a categorization if you want to build go for build a page if you want to reply go for deploy page here is not like that okay now as I said I want to add another task I don't know how to write the code let's consider our Powershell job only see I got a power cell I selected Powershell you are already sure what is Powershell Powershell will ask you either file pyth or inline script where is the path where are the differences I will go for Inlet hello world one more line right host this task was added from task pane okay task parallel let the task so done I have written a script add see the task added there is a lot of code you don't need to write the entire code so task name is Powershell Target type inline you have seen that so I have written script in length there is no file path and the actual script this is a script I have written so why I asked you to keep the cursor here let's say the same task if I keep the cursor here somewhere here okay somewhere here okay keep the cursor here by mistake I am working on something but I don't see that cursor is here now I am going for adding a task see wherever you place your cursor the task will be added there see the task added here actually the task needs to be added at the end of the pipeline but by mistake while checking something I place it my cursor somewhere and I added the task but the task added here so which will show you this is completely wrong your task should not be here because tasks always needs to be under steps but here task was given and Below Pool this is wrong this is wrong so that's why so when you are using classic character kind of thing when you are using this panel to add tasks please make sure where is your cursor okay even if you don't know to write this entire code no need to worry we have classic pipeline kind of task panel is readily available if you want to add any task just to go to the resp ective Place see here okay for a very task there is a line Gap so here also I have given a line Gap I want to add it something let's say I want to I want to run some bad script script basket definitely want to path I can't write inline script that's the problem with basket let it be uh we have some task copy files right copy files see now where is the source folder what is the target folder where is the source folder what is the target folder Source folder uh let it be I will give you random names I'm just trying to add C colon users is a source folder Target folder is C colon desktop click add see the task added completely okay task added completely so if your task is already added and you want to modify same just select the task definitely above the class there will be bedtime settings so generally this button will be in grade mode people will see okay settings is something by mistake added there is no no in our Azure pipelines in the ml this settings is a button guys please make sure that you know the settings button otherwise you will face lot of problems this settings is a button it's not a text so place your mouse there then you will see it as a button you click on the settings it will open the task editing page okay so this parallel will be always hidden if you want to see the panel you need to click on show assistant so this is kind of your assistant to add these tasks if you know ml coding very good you can write directly you don't need to go to this assistant definitely you will not be sure about the Mi task so I'm openly saying even till today I am not clear on some tasks even till today I'm not clear on some tasks so I always prefer verifying this so assistant resource if you see here I will add some task okay IAS web deployed the web name is my site let me check where is my cursor I need to make sure my cursor is here foreign is this one okay my side this one add okay I added this task right but this has many things to manage this has many things to manage so if I add this task my pipeline will fail my Bible will fail why because before I deploy any website first I need to create an app pool for that I need to add another task but this web manage so create website first I need to create a website then only I need to deploy the website so which means if I even add this single task this will fail so here I need to add this task so create website here my site no mandate things add see there will be there will be some specific order so okay now You observe my pipeline is ready how can you confirm that your pipeline is good or bad or if there are any errors so ml has a very good habit it will highlight the errors it will highlight the errors if the error is simple okay there are even there are multiple types of Errors errors are something like kind of warlings information error there are multiple levels in errors okay not error there are multiple levels in issues whether it is warning or error or information or anything so here it is displaying in yellow color which means it is only warning it is not an error okay now if I place this here see here it will displace red color red color means error here it is displaying light yellow or orange so you can observe the color see this is something kind of yellow so this is warning not an error but here it is red color it is error even if you miss to see here see here it is highlighting so if if I am checking here in this section no errors okay in this section no errors but if you see that view there are two highlighted parts yeah see there are two highlighted parts one is yellow one is red so yellow means warning red means error you can see here see this one so this is unexpected property type input so it should be like this okay see all errors are covered the mistake was here but the error was here if I just give a space this error got cleared so that's called indentation if you see here see bad indentation so what is indentation indentation means the structure how you want how you are writing the ml file so where you need to give a space wherein to give a tab everything now if you see I will remove this I mean I will give an extra space these errors will be covered okay now there are noise but I will try to give some name here the entire code is wrong entire code is run so why because this is called parent parameter so if you are giving any parent parameter okay if the parent parameter has values they should not be child values they should not be child values if the parent parameter does not have value definitely it will expect child values okay camel will always be in key value pair structure okay key value pair what is key value pair if you are remembering variables in variables we are creating a variable name and then for that variable there will be a value same here also consider this as a variable what is the value for this now I have given my input no actually there is no value for this there is no value for this then from where I can get the value this is the parent parameter there are some child parameters so all the jail parameters will definitely have some value if I remove this value again it will show error control X it will be empty it will be empty so in my pipeline wherever there is a website name it will consider it as empty value only now it is not showing error but if you run this pipeline this pipeline will fail giving you error that website name was not specified website name was not specified so parent parameter if the parent parameter has a value definitely it won't expect any child values if the parameter does not have values then it will expect child values this is called key value page structure so this is the key and this is the value this is the key but there is no value all the values can be taken from below values that's why we call it them as inputs here also we have inputs what are all the inputs these are all the inputs so input should be from Child only so this is the task name yes there is a value for task there are no sub values for a task right because task itself is single value it has some value sorry single parameter it has some value but inputs there are multiple inputs so I cannot give on this only so I will Define all the inputs here inputs cannot be single value can take any task definitely there will be multiple inputs okay so let me go back yeah so these all are additional added tasks we don't want these tasks from here okay and sometimes what happens once they add the task they will they will actually they will expand or close or collapse collapse is a word so they will collapse the task see here closed closed closed closed so there are four tasks if you want to see the entire task you need to expand sometimes what happens if you are new to yaml you may expect okay this task is a single line task now this task is not single line tasks so there are some steps involved in this task how can you monitor How Can You observe see the count 87 95 so this is 87 this is 95. almost there are seven or eight lines of script is there so if you expand you can see the script you can see the script so this collapse option will be available at the pair and parameter level I told you right this is kind of pair and parameter see here I have collapse button again this is pair and parameter here I have collapse button on the top also see if I consider stage as a parent parameter so I have collapse button now there is no steps at all after 56 you will see empty you will find what is this rajes there is a deploy stage I am not seeing any steps clearly observe the step there will be expand button expand the step so you have all the steps available there okay now these all are unnecessary tasks so I am deleting them yeah okay no for build okay build is running fine but when you are deploying OK when you are deploying AML has one mandatory parameter okay ml has one mandatory parameter which you don't give your pipeline will fail why because whenever you design a deployment okay whenever you design or whenever you define a deployment within a ml pipeline the ml will always expect to which environment I am going to deploy okay to which environment I am going to deploy whereas in classic pipeline okay in classic pipeline if you consider the releases so consider the religious edit see this is the task these are the steps okay this is the job and these are the tasks but where I'm going to run I am going to run this in your environment so stage name is local environment so don't get confused with Stage why because stage is different in the classic Pipeline and Stage is different in the ml pipeline so here I am going to deploy into local environment the same way it expects some value in the ml also but how can you create those environments how can you create those environments here we created a stage if you want to create another stage you can go here add new stage and you can name the stage as test so no I have empty empty test environment created I can add some jobs and tasks but here in the ml where you can create those environments so you cannot create here so that's the reason we create environments here we create environments here see so we create environments here so the same ml pipeline can be used to deploy into Azure app services or to deploy into local servers so when you are deploying into local servers then you need to create environment and you need to add a server wherever you want to deploy it but when you want to deploy into azure you no need to create any servers or ad servers just to create the environment it will be applied it will reply based on that environment definition now let's see a practically so if I go to Pipelines sample website edit see here I am defining so if I go to deploy stage see here I have the environment development now listening so comment so if you give a symbol okay if you give a symbol that means the line is commented which yaml considers that this line need not to be used see here also on top we have some comments these are our comments comments means they are not actual code so don't use this lens here also if you see agent VM image name image name so this all our actual comments they will not be considered now what I will do I will save when I say this it will run automatically she failed sample website already failed why because let's see the error let's see the error see environment is required environment is required if there is no environment your pipeline will not work this is same kind of thing see this environment is required then there is no environment there are no jobs no tasks if I remove this job so if I delete this so where I can even if I delete this so if I delete this see I have artifactory but where you want me to deploy then there is no environments no stages nothing the same way we have created a ml pipeline to deploy something but where I need to deploy where I need to deploy so that's the reason again go to edit go to edit so there are a couple of benefits so when a pipeline the ml pipeline fails you no need to go to the pipeline only you no need to go to the pipeline only you can verify the changes in the reports as well you can verify the changes in the reports as well so what is the error says the error says environment is required which means okay there is a problem at environment level go to your repo verify the ml file so definitely you will find the change okay environment is commented okay environment is commented so you can edit here itself no need to go to the pipeline let's see here okay you no need to edit here you can edit in the reports itself if I edit here then save so I'm removing the comment now commit commit is saving commit now if you see this the pipeline will start automatically see a pipeline started automatically how this pipeline started I have not done anything in Pipelines okay I have not done anything in Pipelines but I modified something in the repo but the pipeline started automatically what is this whenever you make any changes in the repository the pipeline getting triggered automatically so what is this called CA right so this is the additional benefit of yaml pipelines yaml pipelines comes with by default CI enabled if you want to create ml pipeline you should be ready for CI trigger so that's the reason when I modify this pipeline in repository the movement I committed a pipeline started obviously it is showing as CA only it is showing us here only this pipeline will get failed why because first it is running the build job see build stage stays succeeded now it is going for deploy stage the apply stage will get failed why because there is no app service in Azure portal there is no app service in Azure portal foreign should exist before deployment I know because here there is no app service I have removed all the resources in Azure well because those all are paid for me I need to pay for them they are not free so if you go back if you see the pipeline how many stages are there in the pipeline two stages one is build see build stage success another one is deploy stage failed which means your buildup pipeline is Success release pipeline is failed why is at least pipeline failed because there is no app service created in Azure portal simple so earlier there is app service created so the build stage and the release stage both are succeeded release pipeline pipeline comes with by default CI enabled by default CI enabled now if I go for pipelines edit so you are saying Rajesh by default CA enabled how can I disable how can I disable yes see people will say ml is always about coding no it's not always about coding ml also has all options which has classic pipeline so you want to change the trigger you want to change the trigger yes we have option see here let's see these variables guys before going for triggers variables we know how to add variables in classic pipelines right the same way here also we can add variables in camel pipelines just to click on this variables so this is a variables page create a new variable let's have the variable as username the value is okay okay now then I wanted another variable which is password so what is the value per minute one two three this is visible to everyone I want to lock this keep this value secret is nothing but lock button guys how you used to lock in classic pipelines this is luck so now that is hidden now that is hidden okay so this is how you create variables this is how you create variables what if I have some variables in variable group okay here I created two variables here I created two variables part if I have some variables in a variable group uh uh do we have any variables created I'm not sure let me check Library I think we have a variable group yes we have variable group right so I have a variable group are there any values yes there is a value password now I want to use this variable group value in my pipeline what I will do I will remove this password okay save now I want to add that variable group in the ml pipeline this variable group how can we add there is a separate syntax and there is a separate code for that no even I am not sure I need to verify variable groups in the oven so it would directly take you to docs.microsoft.com only it's always better to verify that's that microsoft.com only don't go for any other website you are checking something related to Azure devops or any Microsoft service always prefer Microsoft docs only if you don't get there then go for other values and other websites so you can't create variable groups but you can use variable group okay you can use variable group see how you can create so this is the value so variables group name and the value name this should be added on top guys see here I have variables right here I have variables so I already have some variables defined so what I need to do be something like this group so the group name what is the group name so here I have the group link okay wait a minute okay then group name then what is the value so it's password right now let's verify the syntax so then name name of the value and the value so here name what is the name the name is password actually the name is password it's not the value the name is password the value is one two three this is how you define variable group values now this will take uh let me do one thing we wanted to try this right there is some error okay see there is an extra space you can see here because of extra space it is displaying some error why there is an error structure why so okay let me do one thing yes now this is only warning unexpected property no problem actually these all are common that's the reason so this is a separate pipeline so I don't want to save this leave so how can you define variables in the Army uh let me go through that first so if I go for sample pile click new pipeline okay they are middle consider some some repo some initial Network throttle pipeline so this will be sampled by client so I prefer yaml guys because see when you give full form AML it's easy for you to understand okay here I will add I will remove this task okay this one is not required for me so on this I will add Powershell task okay our cell task inline script so here and host is recruited by ESV it user name in Google password should be places I'm not wrong you just me I am not dude okay yes so this is wrong it should not be braces or how two variables so one variable I will create here itself that variable name is username okay so when I am using in Powershell so this is the script see we are using Powershell you need to specify like that okay no where is this password this password is in variable group what I can do I will Define the variables here variables group so what is the group name here this is our group name the name name is password yes then the value so for now what I will do I will unlock this I just want to see the value add me not one two three save okay I'm not locking this don't own it actually when I specify its variable group I know to specify that I don't need to specify the value also yeah I'm not wrong I don't need to specify the value wait a second guys yeah only two specific so password environment password now this is done say one run press A1 dot now let's see how this goes so we have variable created in the pipeline itself and we have variable in the variable group so we have defined the variable group name let's see how this goes I know permit okay it's in queued state it should get yeah aside them we'll see the script this was executed by rajasem and password oh okay so which means I need to specify the value which means I need to specify the value so here I left empty so I have not given the value yeah I have not given the value foreign [Music] so here I need to specify the value actually it was crucified the value then I can see that no I should not specify the value in the pipeline itself the only displacement of the value in the library where I am missing where I am missing something I am missing here foreign variables Dot so it's always about the syntax s now save so so I am not giving the value let me see what happens in this pipeline okay initial pipeline it is running it needs to get the value from the variable group yes yeah hey what is this man it will not password let me do one thing I'm doing something wrong here edit so I don't want to specify all this okay password environment password save now if I go with the type pipelines run foreign yes so I'm expecting the value from the variable group that's the reason I'm modifying something so it's always about the syntax guys so ml syntax is not that easy so you can see even till today I am verifying the documentation page only yeah now I got the value right so when you have some variables in the group you just need to specify the group name only no need to specify the values just prescribe the group name and directly use the variables in your script okay now I got the variables feature from the variable group so this is directly given in the pipeline and this is from the variable group that's how you specify variables in your ml Pipelines okay now whenever I am modifying the yaml pipeline It Is triggering by default so how can I avoid that let me show you that as well so why because when you are just a minute let's keep on running obviously it will fail I know so Pipelines so the ml Pipelines comes with by default CA trigger so if you want to avoid that trigger how can you can do so here we have the trigger option right so we have the trigger option you need to make this Master to none you need to make this master To None or else if you want to make this change here it is Master right we have options here here we have triggers okay we have options here here we have triggers so here in this trigger you can cancel the ca which comes by default with the ml see override the ml CI trigger override the mlci trigger okay so disable CA disable say override and disable say so now if you see CA was disabled if you modify many times this won't trigger automatically this won't trigger automatically and we have other schedules as same as classic pipeline you can create a scheduled trigger you can create a build completion trigger okay the triggers are similar to classic pipeline only here you need to select override the mlc because the ml pipelines comes with by default CI trigger so even if you are working on Repository you are working on repository and you are modifying the ml pipeline unknowingly if you are modifying the ml pipeline and you are committing which means you are triggering the pipeline every time you modify this file so that will be a trouble for you so for that reason what you do you can go and disable the ca so that way you can modify this Mi pipeline as many times as you want and if you want to include any other branches if you want to include any other branches you can include so these are the branches right the same you can include there are let's see I think how many branches right no branches okay there are many Branch you can include any branches so yaml pipeline also will have all the settings C variables I created one variable right username okay see username I can link the variable groups here okay don't worry that ml means every time I need to go to this pipeline only you can change the repo as well everything is available as classic only okay so now you tell me what are the benefits of ml can be shared with many people and the ml pipeline will be saved as a ml file in your repository so that in future if someone clones your repository they will get the pipelines automatically another thing camel pipeline will have both build and release pipelines integrated together there will be no super no two separate pipelines this will be a single pipeline and the ml comes with by default CA trigger if you don't want to use CA trigger you can disable the ca trigger and at the end of the day camel is nowadays mostly preferred and most very much used pipeline model it may be any devops tool Azure devops Jenkins kubernetes sensible terraform docker any tool they are preferring piano language to write pipelines are to automate activities it not only pipeline any activity if you want to automate any activity so there are ways to write your code in the ml okay guys so try to create multiple pipelines in ml and try to deploy so for deploying environments and mandatory so if you want to deploy local create environment and add the virtual machines if you want to deploy Cloud just to create the environment and use the environment in near me why because you can't add any resources in Cloud environment okay so with this we are closing our session on yaml in our next session we will see about agent pools how to create Asian pools and how we can configure agent pools okay yeah thank you guys

Channel: Azure DevOps by RajeshkumarM

Views: 6,746

Keywords:

Id: p3yFmTaRl7o

Length: 62min 55sec (3775 seconds)

Published: Sun Nov 27 2022