Azure DevOps Build Pipeline (CI Pipeline)

what's up YouTube in this video we're gonna be looking at as your DevOps build pipelines now in the previous video we talked about as your get so I'm assuming you have a repo in aggregate in this video I'm going to be talking about automating the build process and the build process could mean many many things but in agile devops is just called the build pipeline or the CI pipeline and so that's we're going to be talking about today in the course of this video I'm going to be talking about a whole host of things such as the hosted agents or custom agents and variables custom variables predefined variables build tasks and on and on so this video is gonna be really interesting for those of you who are looking to build our automate your build process in Azure DevOps all right now the project for today is decently complicated so it's not a toy project it's a fairly decent-sized project and it has some complexities and in the build process which is why I chose to use this product in today's video I'll make the complete llamó file which is the definition of your bill pipeline on my git repo and I'll put the link down below in the show notes so here I'm going to be discussing the different pipeline parts of the mo file and go into depth wherever I think I think is necessary just to give you an explanation of what this project is or what it comprises it's a dotnet core project but it has a database project which is not dotnet core is dotnet framework and my tests require the database which implies that doing my build process not only do you have to build these different projects that are from different frameworks but also I've got somehow managed to deploy a database on the build servers in Azure DevOps and connect to these database this database and then my tests can access this database and function completely so there that's the complexity is sort of a multi platform if you will dotnet core internet framework was non multi platform is multi ESD cave but it also that complexity is adding some twists and turns and then the database itself the fact that you have a database and you require a database to automate your build process that makes some things a little more complex now in Azure DevOps the bill pipeline I could sometimes say bill pipeline or the CI pipeline is one of the same thing in agile devops I call it the bill pipeline even though they're talking about continuous integration they keep saying the pipeline so either way I think they're so synonymous in this particular case so what is the purpose of a biddle pipeline essentially you're trying to automate your build process now in some cases the build process may be really simple they just build your project and you're on your tests and everything's good because your system doesn't have dependencies that may be generally missing on a clean machine by clean machine I mean a Windows machine with Visual Studio on it that's it right if your build process requires other software I'm not like about nougat packages and things I'm talking about other things let's say you're using rabbitmq let's say you're using an azure service let's say using a database let's say what if you are building your system on Azure DevOps but the dependency is on premises which means you need to be able to come in to your network from the outside from agile devops to your on-premises services all of these features and capabilities are provided in agile devops and I'll go through each of them one by one including if you don't want to use the hosted Microsoft hosted agent which is a hole in agent bill agent on Azure DevOps you can actually host that you can have a custom agent on your premises on your local machine I've done that is fairly simple to get started I was actually pretty amazed by that whole process it's been a while since I did that but that was certainly an eye-opener Stu how easy it can be to have an agent on-premises and have a pull request being accepted on a jacket that triggers off a bill that calls back into some machine on your local network and starts the build but in this video we're gonna be only talking about the Microsoft hosted agents and show you the enamel I'm not sure the Yammer will change for any reason it's just that the agent will change in your Yama file and everything else should be the same okay so to get started I'm going to just navigate to my azure dev ops account and so this they call this the organisation and then these are called projects even though in in from a visual studio parlance perspective this is not in a solution it's even higher up than a solution but they call it a project in either DevOps so I went to this project in the repose I can have n number of repos here I only have one for now but this repo is synonymous to a visual studio solution so just keep that in mind so that you don't get confused because the then four-seater terms are the same but have different meanings for us in video studio versus the NIDA DevOps so this is a solution and as part of my source code I have the CMO file now the extension is yml the acronym is YAML and it used to be called yet another markup language but it seems the folks at Yambol have decided to call it not yet a marker language but yeah mo ain't a markup language I call it a mo it's just simpler so this is what the llamo file looks like what can you do with this UML file the whole host of things you can do with the co file it seems a little complicated as your dev ops didn't used to support emlv forces a relatively new feature in agile devops it used to be a more visually oriented way to build a pipeline or the configurable five line now it's mainly AML base but you can resort to they what they call the classic little pipeline alright but we're going to be talking about just the Yama file so if you go into your repos sorry into your pipelines you'll see the different pipelines you might have configured for your repos and if this could be two different repos in this case it's only configured to the one and what branch these these pipelines are for in this case both of them are associated to the master branch so you want to create a new pipeline you've come here and then you pick the source of your source code or you could switch to the classic editor in the classic editor as I said things are a little more visual so you're gonna do the same thing you gonna choose the source and once he chooses the source the next thing you choose the kind of project if you will so you could say it's a dotnet desktop app or a asp.net application or asp.net core application or a asp.net for.net framework application exit etc so this project is an HP dotnet core dotnet framework application I hit apply and what it does is it starts builds up this pipeline it's pre-configured certain tasks and I'll explain what these are the tasks and things but for this specific the build solution part there are various variables or parameters you can set up and each of these parameters of course would mean something to you in your project or in your solution and I'll explain to some extent what each of these tasks are and what their parameters means so you can get a full sense of those tasks but also there are a whole host of bill Towson agile devops we will be covering a small portion of it so it's something you have to experiment with or Explorer so now even though this is a visual way of adding tasks to your pipeline you so you can look at here these are this is the pipeline and that's the name of the pipeline and I agent pool and so on so forth and then for each agent is a job and for the job there's different tasks all right so if you wanted to look at the variables for your pipeline that I set up over here some of these variables are sort of pre-configured and some of them are custom so there are in agile devops as I said there are pre-configured variables that you can use in your bills or in your tasks and you can define your own variables and you can use those as well in your tasks there's a whole host of a whole list of predefined it's not just these but there are some of these are just kind of there by default all right and another interesting one is the system dot debug which is typically the set up to false going back to your pipeline you can also add new tasks or for every task that you click on you can see the parameters of this the variables of parameter that have been set up for each of these tasks right I won't spend too much time here view the the equivalent of that is the llamo you can look at any one of these jobs and then get the ammo for that here and this essentially is the ya know for everything that we see here on the left hand side here you can see their steps in this toss and as the nougat tool installer which is that there's the nougat restore happening over here then this vs build over there and so on so each of these visual elements are essentially converted back into mo alright so you can copy doesn't start from there or you can start I'm just gonna go back to my pipelines I want to say new pipeline I can pick my repo in this case is from Azure gate and that's the repo and then I can pick a speed on the core framework and that's the yellow right now from here you could also visually go back here and say okay let me search for let's say for example if you want to look for msbuild so you can search for ms bill and you can pick this task and drop it in there when you click on it actually puts it where ever the the cursor is you can specify the different arguments here when you say add is this going to go into the location where your cursor was right so there's many ways to get there some as visual some of this hybrid either ways I suggest if you're starting off even though it may seem a little complicated just start to stick with the animal since that's sort of the future on either DevOps it's normal to go back to the visual part it's when I eventually deprecated the visual side of things but this is semi visual anyways here and that's how you would create your new pipeline but I'm just gonna skip this step go back here this is not gonna save it so I've got a couple of pipelines set up here once you set up a pipeline and this is essentially showing your pipeline has succeeded building this illusion in this case is just both these pipelines are actually working I'm gonna pick this one here and then I can all just go back and I can edit this pipeline like that these menus if you're not familiar as you DevOps the menus are hidden in some ways and you may not know to look here or there for certain menu options so they have them set up over here see by the 42 SEC it don't really move your mouse over and you got these menus over here to allow you to handle pipelines so I'm going to go here and then what I'm going to edit I can even run this pipeline directly from here so it starts a whole nother build process or I could go into edit to look at the llamo so as I said there are of a build pipeline or a llamo file is comprised of multiple steps and each step has multiple tasks it also has jobs and a job has multiple steps and steps have multiple tasks and so on so it can handle quite a bit llamo can but typically in our bill pipelines in our release pipelines we don't need to get that complicated so I'm going to step you through each of these tasks that I have here explains all the pros and cons and what you should look out for if that applies to certain tasks but you'll get a sense of what is going on in these ham llaman files so essentially we're gonna be picking a hosted agent this is a virtual machine in this case hosted by Microsoft so they call them Microsoft hosted agents and these agents are virtual machines depending on the agent you pick they have various software pre-installed on them now that's not to say that they have software that you may require but the East come with a certain set of predefined software such as Windows if the pick a Windows agent Visual Studio nodejs this is a whole list so if you look at the different agents if you go here there's a website for all things as a DevOps related and additive ops pipeline related so you can see here of things some here now in the Microsoft hosted agents there's a list of agents here and I typically use the in the previous editor in the classical days to be called Windows and 2019 and Yama we say we call it Windows latest now cause this might change today's Windows and we just to do 2019 but tomorrow could be something different another version of Visual Studio and if you link here if you click here you will see the different software that is pre-installed on this VM now the reason they have it of course is your bills may require other software or if you need to install additional software let's say you got your own I don't know third-party thing that needs to be installed on your bill servers because your software didn't relies on it then you have the option as part of a task to say okay first go install that software then go build the solution base you have these various tools here such as our powershell and a bunch of docker images and of course we just to do twenty nineteen enterprise right so every agent every hosted agent has different sets of software included so you might wanna look at each of those to decide which one you want and you also have the option to use Ubuntu or Michaels as your build a Djinn in Azure right so those are those are the agents if you look back here the first thing I'm setting up is the trigger for the build he's the master branch and we have set up the MAS branches anytime a pull request comes in and the code views approved then it automatically kicks off a build and that's a choice you have whether you want to make that automated or not but those are some of the features in agile devops you can also have emails sent out to specific people or all the entire team every time I build starts every time I will succeed every time I build fails and so on services lots of all the order automation available and notifications available to these automation automated steps in agile devops I'm not gonna go into the details of that but those are easily discoverable and I don't think deeply people get stumped with the in that area most people get stumped in the yeah Mel file and the build process and how to set up the build pipeline so that's the focus of this video the trigger can also be different branches maybe you each of your devs is creating a new branch when they start you could set up a bill for each of those branches so that when that branch is the pull request is pulled as accepted it builds on that branch and that pipeline could be different for that branch versus the master branch and sauce there's lots of config ability and variations available in Azure DevOps the agent they call it a pool the agent is the windows latest alright and that's so this is a pool but then this is the variables so some variables are predefined some and you can add your own variables here so you can give it whatever name you want and any value you want and then the way you use the variables in your llamo files is dollar pen and then the build so get the variable the name of the variable it so any time you want to use the variables you just put in the dollar and brand put the variable name in there and close print all right now these same parameters are available through a DOS prompt so let's see you need to as part of a build process have a dos a batch file run for whatever reason maybe there's a feature or functionality required that's to be installed or set up on the local machine that requires you to use a batch file see you can run the batch file as part of a step or a task in the build pipeline and maybe in that batch file you need to access some of these variables you can do that as well you can do that from PowerShell you can do that from batch files you can do that do that from bash scripts and on and on so the same variables the names change a little bit for the same variables and their values are available not only in your Yama file but also in batch files in powershell scripts and in bash files that so we have some of these predefined variables here again I'm just on a Microsoft page here I'll put some of these links on the in the notes below so you can go directly there it's easier to go searching for it but it's very valuable the information in these various pages on Microsoft with regards to either DevOps I suggest you give it a read you'll gain a lot of insight as for how things are working in how to exact configured and how to even debug your build process I'll talk a little bit about how to debug your bill pipelines but I'm not going to go into a whole depth of that so the websites will the pages here will actually help you with some of that so there are predefined variables such as system that access token system debug agent build directory agent or home directory and so on and you might need some of these variables when you're trying to recive determine the path so to see you need to do something in a DOS in a batch file but what's the path like where am I at and where's this solution and what's the path for that solution so all these variables will give you those in that information so you can configure it or make your powershell scripts or your batch files and make them actually work being independent of your visual studio parts so the bunch of predefined variables and there are also variables that are build pipeline variables and some are release pipeline variables so today we only talking about the bill pipeline variables so they do distinguish between the build pipeline variables and release pipeline variables and of course custom variables are things you have of iran now if you're trying to use some of these variables from how any of these variables from your less a batch files then they become capitalized and of course in typical dolls fashion they'll have percent signs on either side of them so the dot gets replaced with an underscore and in daus you would put percent signs on both sides if it's PowerShell you put the E and V dollar E and V open friend but the name changes in all cases to uppercase underscore so where there's a bash script or a PowerShell script or a da script they all change the name to uppercase and the dark gets a paste with the underscore and of course the syntax is going to be based on that that the DOS file of the batch file they of the bash file or the PowerShell script where that dollar comes a deep for D and V and so on so forth so that's all part of that environment but all the variables are available to you there so in this case the first thing I'm doing and this is just for for demonstration purposes I don't actually need this task in my actual build process but I'm showing you that there's a batch script task you can simply copy this and go search for this in Google and you'll get a whole host of information or you can look at the catalog now they show you here the build and release tasks there's a whole catalog here for build tasks are different from released to us and the entire catalog you can look at by expanding things out and you can see a whole host of information here so if I want to look at dr. Corsi Ally this is one of the tasks we actually use in this built house I'm going to show you and so that's what is described as in the in the yellow file and these are all that possible parameters so that default values some of the valleys are optional even though they're optional they'll also specify the default value like in this case it's going to say publish test results choose the default but it's optional you don't understand so it's going to be true if you don't want it or not to say it's false so each of the tasks have a whole host of documentation if you know the name of the task you can just copy it and search for it and you'll end up on a fart like this on a page like this I mean right so going back here and then so what would happen in this batch file is that it would execute another batch file and I'm saying if there's any errors in this batch file that it should fail that this should not continue on you can look at the batch file here so we have certain variables here the build configuration and build platform if I go into the repo you'll find there's a batch file over here have a look at the batch file this is what it looks like so it's essentially pulling out the variables the different predefined variables that we have access to and I said in DOS you're gonna say percent and then the dot gets converted to a underscore right so I'm just testing out the different variables and their values in this case I'm also saying I want to see the folder contents and this sort of thing comes in handy when you are initially setting up your pipeline because you may not understand the folder structure and you do go to folders up from where you are to get to the another file or the two folders down and so on so forth so this sort of saying the agent work folder the bill directory is the source directory all of these things will help you understand the environment on that build machine on that VM and then you can figure out how to get to this stuff you're trying to get to so this is a nice way to figure that out so I'm going to leave this in the file I put up on github so in case you're starting out and having to haul some of these values are going to help and you're going to see when there's build pipeline runs the results of that and we will check it take a look at that as well okay back to the pipeline okay so now the first thing we're gonna do is build our solution now in this case I'm using vs build and I'm telling vs be able to restore all new get packages and build the solution so the solution is whatever SL and father can find in that folder if there are specific multiple solutions in the same folder in your corner to specify which one and I wanted to be a clean I wanted to use the maximum number of CPUs available and I wanted to use the upcome build it as a x64 architecture the next step is - now if you remember in my case I said I need before I can run the test I need a database so in this case it's sequel server or sequel local DB and that's another feature that's pre-installed on this on the virtual machine which is sequel local DB which is sequel local DB which is the command line for sequel logo DB and that command line allows you to create a new instance give it a specific name spin up the instance and then you can also if you have a backpack which I do in my case I can hydrate that database on the build server and then I can run my tests as we're going to be doing over here so I use the sequel local dB and it has the command-line features I'm saying create a instance and instance Windus by this name and then start up the instance now there was a time in a DevOps where even sequel local DB was not available so it was quite a bear trying to get that to work by I'm glad Dave now giving us the feature where sequel DB is pre-installed on the machine when I logged the complaint of the bug reporter then they said it should be there as it was not and then a few months later it appeared and so now I'm able to make it simpler so so I'm using the command-line here to just do that right now once this is the database instance has been created in the database instance it started I can then use a tasks that they already have now call the SQL DAC back deployment all machine group and that's the full name of the task and it requires a back pack now that pack is Lee is the outcome of a build of a database project in Visual Studio just like a DLL is the output of an assembly or he exceeds the output of a application executable the DAC pack is the build version of a database project so it's not like as weird if you haven't used that and that's what a dye pack is and so you see here at a figure out how exactly do I get to my backpack and so initially of course when I didn't understand how this was working in as a DevOps I had to use that test or that kind of thing to figure out where is my directory and then how do i navigate to my backpack from where i am and so that was the base austerity and that's the path to my pack and then i also have in my database project and this is not necessarily for i DevOps this publish XML file which is something you would have when you're using database projects in Visual Studio is to deploy the database let's say on a local machine so there's a published XML file and you can have n number of these publish files so I already had the publish file so I'm just using the same publish file to then publish my database to that instance that was just stood up and the reason I do that is because I also have a post deployment script which sees the database with certain information data like lookups and things like that so I need to have this XML file executed or the publish script executed so that it does the the builds of the database and also does runs that post-deployment script again this was not required just for as a DevOps I needed an e or my local machine and to run my tests and things on my local machine so it was already there so I'm just using the same thing so that's the published profile so now by this by this step the database instance has been set up and started by the end of this step my database schema has been hydrated in the database and the database has been populated or seeded with certain data and now I can actually drop my tests so in order to run my tests in this case because it's kind of like a hybrid thing I'm not using msbuild even though I could have used him as build for this I'm not sure why exactly I start to using.net core CLI but I've also used him as build for this degree doesn't matter but dartnet course ela has different commands one of them is test the other command is a bill and package and a whole host of other commands so when I do tests now the default behavior is in.net Corsi alive we need to test is going to do a build but since I've already done the build in the previous steps I don't want to do a build again in the azure DevOps you are charged for how much time your bills take and I think they give you and I please don't believe what I'm saying here in terms of the actual numbers but I think it's 360 hours or 36 hours or something like that of free build time across release bills as well as Bill's right your CI and CD so of course you want to make sure your build pipelines are optimizing that they're not wasting unnecessary amount of time so I don't want to build it again so in this case have they have a parameter that you can set to false so that it doesn't rebuild the the solution that's already been built now cuz I have to build the solution first because I need that back back so I can't just do a test where it starts to build and launch the tests because I don't have a database yet so I would like to break it up in multiple steps so that's this step which does the actual running of tests and then and some of the arguments here to the tests are things like the run settings file this is again a visual studio feature I'm using it here because I wanted to ignore from a cold cover standpoint the test products I'm not interested in getting cold courage for the test projects only need the code coverage for the actual project and then the switch for the lover which is the actual coverage file code coverage file and then the argument that says I want you to collect code coverage so these are typical msbuild arguments nothing specific to this task but they're also available on this task so I put these arguments in there in the argument section I'm telling it if this task actually needs dll's so or project files it's not going to see every teleport for the files I'm sort of using a wild card here to tell it what project files to look for and once the test run I didn't ask her to publish the PDB files to a certain path and then I wanted to copy all the files from a certain folder to a certain target folder and then publish the artifacts which is essentially just a staging area where the next step the release pipeline can kick off from there so your release pipeline will say pick up the the artifacts from this folder and then push it out to the servers and either right so this is the build pipeline so I'm gonna run this pipeline now there's a few ways to run it and there are a few ways to debug your solution into the polymer you can come in here and you can say run and there's a few options here one is you can enable system Diagnostics that will give you some additional help or information if something is going wrong and get a full detail of what's going wrong there are logs at the end of the build even though the build fails you have access to the logs that should help you figure out what's going wrong or you can use this variables thing here and you can have a variable and you call it system dot debug and you could set the value to true now the motor turn this off after yeah I think it it remembers it I'm not not currently certain anymore but nonetheless if it's on you need to set it off again but I think this is for the specific run so if you set this to true then you get additional information at the time of doing the build so when the bill fails you'll have a whole lot of information so between the system the debug equals true and the the previous option here that we had to enable system diagnostics and the logs you should be able to handle or figure out what exactly is going wrong the complexity comes only of course if your projects are super complex meaning they're not the normal structuring if things that are kind of out of the path then ordinary root from the root of the solution onwards with a crap out of that root or you have a bunch of dependencies that you install and that complexity might add to problems but by itself it sort of just works nice if you follow this typical for low structures and things in the general dotnet community stuff just works but as I said in my case I needed the database project and so on so I'm going to pick any different options here I'm simply gonna run it so I could run it from here or if I went to my pipelines and I clicked on this here or just from here I can sit on this pipeline so there's many ways to run the pipeline's I'll do it from here and then come back here so same options I can enable this or I can use the variables I can also pick I just I guess on what variables - I'm never actually use this but you can pick different stages now I said the run it's gonna cost you and it's not the death is so reasonable I mean I've never had never paid any extra money for doing all my bills and I do a whole lot of them but of course you want to be cognizant of the fact that and there's a limit to how many free minutes or hours you get for a free account a nagger for private git projects so my bills typically take about four and a half minutes or so so the way it looks is and this is almost real-time it's not read in real time but it's almost as you can see stuff is appearing in this command window window over here it's just a looks like a come on window so it's not actually a command window you can type you can't write commands in the window here but you're seeing all the different jobs you're seeing how much that's all it is actually tasks and how much time or in seconds or minutes each of these tasks are taking so the bill takes approximately I think like a minute I believe and then it's going to create the local DV that's it apart so I'm going to forward and then we come back and examine their port or outcome of all of these different tasks so you can see what that is like all right you all right so this bill tasks as bill has completed successfully as you can see all of them on green here if we click on each of the individual tasks here you'll see the output in DOS command window style and output and this would be interesting for those of you never seen this before this is that batch file that we ran here so I'm saying system door debug value is false because I didn't set it to true so there's nothing there hollow wall from hosted agent ID is 10 agent work folder is that and then the contents are all of this so the different directories and so on so us this is going to sort of help you decide or figure out where exactly are you situated minute on these bill tasks and how to get to the actual faster you're looking for this is the bill agent directory and then this is the bill source directory this is essentially mine the route of the border here so you can see the ml file is here the batch files over here and so on so forth so once you get to this point of course you can navigate anywhere within your own structure because you already know that structure that's kind of one way to debug the way to debug is to V the logs here this gives you a lot more detail any one of the jobs you can click on and see the the logs now here you can see there are there's a warning for the build process here so this is the full set of tasks here now if you go back in here you're gonna see that this job succeeded and along with this job it took 3 minutes and 32 seconds and now for the jobs output I can look at the test results over here so you can see in this case there are 40 tests all tests pass $1 percent pass rate and the tests of 28 seconds and 952 53 milliseconds now in my case I also have court coverage now court coverage is not something they have a visual description for here but I can download the coverage file right so I can actually download it and I can open this file it opens visual studio but it doesn't it can't show you the source code so what I'm gonna do instead is I am going to close visual studio again just to see that file where is it putting it and I can say open it show in folder seconds see that's the the file second now say open with well let me open visual studio here with the same solution so it gets access to the source code you could even do it the other way where it'll you dig into is going to say hey I don't know what the file is and once you point it to write file or you find all the other files but this is another way to do it you can simply drag drop this file on the wheel just to do here and I should show you the code coverage resolves so you can see the court coverages 99.1 2% and if you want to take a look at what's missing what's wrong here I can navigate to this location to find the culprits and then I can actually go to the actual source code so you can see here that this line is partially covered because in this case it there's a conditional statement that's not the problem with this question mark DARP you know if you start using this it's gonna be very hard for you to get the court coverage now that's not to say that you shouldn't use it when you need to but that's one other sort of downside of that but as I can see here this is a partial coverage it's not a red when it's red it's a full coverage basically all of these situations are partially covered because it's going to be virtually impossible now for me to create a scenario where the DB command is now the prime action is no and the so on so forth so anyway so that's the code coverage there's no visual representation in in some ways that's a good thing because visual studios code coverage analysis is super and it's very very informative and useful and you can navigate to the actual code so I can't imagine the same feature and we in agile devops or webpage giving you the same functionality so you might show you the code online but here you are on in your solution on your desktop and you can kind of navigate where you want to and fix it and start working on the missing in this case code coverage and stuff like that so this takes us like a full circle from starting a new pipeline understanding what pipeline tasks are variables our agents are having the run and then looking at the final outcome of that task now if we go back to the actual task here you can see there this task build task has had so many different runs so you can go at any point in time you can go back to any one of these pull requests that has a build and look at the details maybe there's a failure while it fail or how much time did each of these stakes of this one took duty takes about four minutes four and a half minutes today it took us 30 minutes in 38 seconds so it's faster all right so this has been a useful and interesting episode for you all I hope you enjoyed it if this has helped you please give me a like please subscribe please hit the notifier button and I will see you next time

Channel: Shiv Kumar

Views: 4,497

Keywords: Azure DevOps Build Pipeline, Build Pipeline, Azure, Git, Azure Git, GitHub, Visual Studio, VS, C#, Azure DevOps, DevOps, Version control, .NET, Asp.NET Core, ASP.NET, CI/CD, CI/CD Pipelines

Id: qBa5vf1joWw

Length: 40min 8sec (2408 seconds)

Published: Fri Feb 14 2020