(Simple?) Real World Azure Pipelines YAML Walkthrough

look cicd is so easy you can just click through the portal and we'll pre-populate your project with an azure pipeline and it'll do all the steps and deploy great right yeah that's useful for like a week and then you go into production and you're like nah this is way more complicated hey there my name is julie i'm an engineer at microsoft and today i want to walk you through some real world pipeline yamls and how you can optimize and reuse code with yaml templates how you can do sort of environment specific tasks i'm going to show you my workflow where i automatically generate change logs and sort of tag my projects which are also signed and pushed to get and they trigger all these different workflows and um importantly the docker image that is going to be deployed is going to be scanned and tested um when it goes to the main branch and when it goes into production actually it's going to grab an existing image right so we're going to do a little bit of promotion um this will come across as a little bit complicated but actually this is a simple pipeline in future videos i want to add more things like stages kubernetes um but let's start with something that already exists and is kind of easy to follow okay uh final note before we start i'm going to use azure pipelines and not get up actions it is april 2021 and github actions are nowhere near mature enough uh for my use cases right so security is really important and uh for this video templating is really important somebody posted a month ago yeah it still doesn't exist um and basically you would have to publish a distinct github action in order to reuse code no thank you okay so let's look at the original pipeline very briefly um it is one long yaml file and it has two stages and these jobs all run in parallel the idea being that they should run faster if some of the things are running in parallel and then we have a deploy stage and so how it knows to deploy let me just open up the yaml file development or production it's based on the branch names and you'll see there's lots and lots of variables set at the top and they will be replaced uh if it's a release branch and there's only one step that says basically um deploy and so um i feel that way about this pipeline it's kind of too long um and i made this i think the first time i was really learning sort of azure devops i'm trying to give it a chance and not just say i miss jenkins and although i am happy with parts of this that i got to clean up very well i think it could be better and that's actually the goal for this uh video i'm gonna show you now how i do releases especially with versioning and it's pretty cool with standard version um so i'm in this repo i'm on the main branch right now i don't have branch protection on it because i'm going to push directly to main as an admin i'll set up a projection later for the production branch but for now let's look at the git log and you'll see that actually i have everything on my remote already that's why you see origin slash main and the last two commits you'll see that they're prefixed with a fix and then there's a feature as well and so because the last tag was zero seven one right the next one will be zero eight zero it's not gonna have a dot one appended to it despite the fact that there's a fix here because there wasn't a release in between so what i have is something called standard version and i can just run standard version because i've installed it globally on my machine and release as minor and i'm going to sign this and what it's going to do is take based on the git log it will bump everything and generate a change log for me and i need my key okay so now i have my ub key in the monitor because i'm out of slots okay so that's all i did and uh we can look at it in here because it's a little bit easier um i think preview and you can see that it generated the change log automatically for me and linked it to the commits super cool um you have to be pedantic about your commit messages but you save a lot of time by doing that so it also tells me to just do a push follow tags origin main and let's do that again to get the credentials to actually be able to push to get.com github.com rather okay so i just made a push and so i'm actually going to expect the pipelines to run and this one is for ci so it should build the dev container and this will be interesting so what i did do in terms of triggers is that you'll see that this is being triggered by the tag being pushed and you'll see that it actually skips the ci the test i don't need to do that because the assumption is that it's going to already be tested although this one was caught first right you can see here the git shaws are the same so this will be the one that will actually be doing the testing uh when it's sort of uh passed um now that i'm looking at it's probably not ideal theoretically this could fail first and then the tag will go in but that's okay um for now for me in my use case you probably want to do it a little bit differently okay so if we go back somewhere there's a release right and i actually didn't show the various apps so i will do that very briefly right now and you can see that the dev version is running 0.71 and the prod is still running 0.70 which still has a bug um this says nine seconds because it's a bug and you can read the docs more and so this one's actually more appropriate um and it deployed a few minutes ago and so it's only three minutes old so this is uh working on depth so what i want to do is make them both actually go to zero eight zero um and if this is still okay it's still deploying that's why apparently let's go and just double check okay the asset pipeline um since this said it only finished actually 40 seconds ago that's why this is old um so eventually this will cycle through i don't know if i should just hit refresh refresh over and over but this should eventually cycle through and say zero eight zero okay so this is what it looks like now i actually have two pipelines one is called dev and one is called production um and if i come over here you'll see that actually it looks very different from before there's a lot of subfolders with different things and i'm going to switch over to vs code because it's a little bit nicer to show you the difference so like my usual pedantic self everything is sort of documented but i'm going to walk you through that right now so um you can see that i have some variables um and they are actually a lot um and i've moved them over to a separate file so i have some static variables here which i could put in a variable group but i don't like hiding things right i like it as close as possible to get so when i'm debugging i can just look at it here instead of going and trying to find stuff then in here and you'll see that there's actually lots of random things that i have in here that i should probably one day clean up um it's good use case if you have things that are sort of shared but this particular project doesn't have that so like in the previous pipeline all this like uh really wordy um code to just figure out if something is production or if it's a git tag or if it's a fork etc i've put in its own sort of separate file this is reused and it's just easier to see one thing that is new that i think the old pipeline didn't quite have is um this new variable called is trusted um ci so if you've watched my previous videos you know i'm a stickler for security and although some platforms have built-in methods such that pull requests or especially from forks don't have access to your secrets i like it doesn't hurt to check yourself right so i do that basically what i want is that only events triggered by humans um will sort of deploy right or have access to those secrets so if it's a fork if it's a pr or even if it's scheduled that's me being like maybe paranoid i don't wanna yeah give it access to stuff the other thing that is new is down here you will see that there is a variable called um image tag that depends on whether or not it is a git tag that's actually pushed so the default is i'm just going to use dev as the tag for the image but if it is actually a tag i'm going to take it from the git tag that was pushed and just removed the v so if it's v 1.0 you would see like 1.0 um what you will see is that these pipelines so dev or production um depending upon uh yeah which one it is um it will also then uh load uh variables right so these is all suffixed with dev for dev variables um and yeah so that's it for variables so i keep noticing random things and i'm just gonna publish this video i'm not gonna fix it and then re-record um so initially this was named actually ci for a while and um i had here like okay is it main is it production etc um but actually now that it is only um dev i don't need this like at all and there was something actually interesting and i think you see in the production pipeline so uh when i was originally splitting it up like this what i painfully discovered is that i can't put this in the global scope where over here i had to put it in the stage scope and so my guess is that it has to do with all these different sort of conditions and how they're processed so when you see the curly braces um that is actually evaluated at uh template build time when you know azure pipelines figures out okay let's take all these ammo files and what do you actually want so what i'm gonna do right now and i'll push this afterwards um is i'm just going to get rid of this uh variables i don't need a stage um i should probably cut this out of the video but i will forget if i don't do this so let's do that dev is this i don't ever need this and i'll explain in a second [Music] um actually i thought the animal so i don't need this either um so why i ended up actually instead of ci uh yamo and um cd yeah why i have dev and production so if you are very astute you notice that i'm recording this actually almost a week later um than the first bits and that's because yeah i only work on things um every now and then anyway so um if you've watched some of the talks that i've given on ci cd that my preference is actually almost always for a yaml file per environment it's just easier to see like what's going on and one of the reasons why the ci uh yaml eventually became devdyamo was that i actually did want to deploy to dev stage right and then it was like it felt weird should i put that in the ci yaml didn't quite yeah feel right to me and so i just put it um here as well in this formerly ciaml now called debiamel um i split up the app service deployment into its own job and you'll see that there's a few sort of here most of them are actually not really reused but it was just easier to keep these files the dev yaml and the production yaml slim and so i moved a couple things over into uh jobs so let's look at um first kind of okay what we're doing so in dev um i run some tests and actually if you go to production you'll see that it's very slim and like the only thing it does is actually deploy it doesn't run any tests and it doesn't um build any docker images because basically it's going to grab something that already exists in the container registry and assume that it's okay right that it's secure and that it works because yeah tests from before so going back to dev um tests should be no surprise it's pretty much like what we had before um and if we go back the next stage is the docker build stage okay so this is the docker stage which uh yeah was a lot of fun um but let me first explain the build and scan job i wanted to test security or test for security issues before i actually pushed it to the azure container registry azure container registry has a security scan but obviously we need the image in order to scan it so you have to push it first and so it's reactive basically like security so this is proactive if this fails right and i'm using the snake api then i don't want to save like whatsoever and then of course the whole pipeline fails and never gets saved so once i know it's okay i actually want to build this image tag it properly and push it to azure container registry so the next part is the whole build and push there is this built-in task it like basically takes um as a multi-line string various tags um it totally did not work for me um and like just put in build numbers etc and i think in part because i have this sort of if here um whether or not it should include the tag because remember if it is a image intended for production i'm not going to use the git shot on top i'm going to use the version number from the git tag itself so i needed a little bit of logic um to be fair i think the build and push does a couple more things like add labels and a bunch of other things but um i've always like not really liked this task um because yeah it's just weird for me to have inputs you have to tell the repository and then the various tags like kind of i don't know i'm kind of old school and this is super easy and i do this on the command line on my computer so this works yeah it works um ultimately when i cleaned it up i only had one image tag but still i just was like i don't need the build and push tag um i know how to do that myself so once that is done then i log out and this is the part where i lock the production image again to keep it immutable because it's running in production okay so going back to dev.yaml you'll see that i have a deploy stage because if it's the main branch i want it to actually go into development and so you'll have a bit of reuse anyway of certain stages and steps in your pipelines and again instead of doing ci cd i ended up going with the environment and these can run in parallel the app service and the asset pipeline now is a good time to actually explain what that is so if we come back here we have i added a wrong page main repo i added a architecture diagram of what this looks like so the web app itself it has all the images and css that it needs but a common pattern is actually for certain things that are static like images and css you can actually load them from object storage and then to be really like sort of icing on the top you can put a content delivery network in front of that and so this application has an environment variable that will that it uses to determine where do i pull these assets from and you can always use the domain itself if you want to and this part again can run independently because the assets themselves are also in the web app so going back here let's just look briefly at what that looks like super simple it doesn't really do anything um it just deploys the container container the image tag might change right so it might change using the version from the package json and that for me is the way that i could deploy to production multiple times using the same image version right if i only react to the git tag event i'd have to override that event over and over and over and so for uh production i will just grab from the package json the version appenshaw i just decided to put it somewhere else uh where is it um i can't read a pen sha so and what this does is that it uses the um the git sha and the built-in variable from azure pipelines is like that long nasty i think 40 character string i only need this bit so i think i submitted feedback ages ago that we need to offer the short version yeah they didn't do it so i do it by hand and so i don't do it when it's production or tag this is a bit me being pedantic um and uh yeah so both when i build the image i need to tag it with that shot and when i deploy it i need to know like what that shot is so once i've done that that works so this is the production pipeline pretty much the same um the only thing is again we're not doing any tests and we're not doing any docker builds because what we're assuming is that we're promoting an image from a previous step so i'm not rebuilding it and yeah that was easy the last thing i want to say is that it's not done right so if i run this command i want to see what tags i have in the container registry and what i had done before was that i would just constantly use the dev tag that also has its consequences because you're constantly overwriting it there's still an image there that doesn't get deleted right and azure container registry you calls them i think untagged images which you'll see by the manifest anyway um as part of this refactoring i switched over and like i said this is the built-in variable which i think is nasty i like them yeah short and sweet so i use this um regardless of which git sha you have you're going to have to clean them up so by hand obviously i'm going to delete these but in production you're going to have to come up with a strategy to clean these up so that's going to be some scripting and maybe i'll do it maybe i don't microsoft pays for this account so i'll just clean it up every now and then and it's all good um but as i tell my customers i don't give you free fish i just teach you how to fish and i think there's actually already a lot of free fish in this repository so figure it out clean it up and um yeah i think the last thing i want to say is that these are very opinionated pipelines this is kind of like my workflow you saw that i automated change logs i pushed to production i applied some git protection rules but you also saw that i kind of just flushed push to the main branch because i was trying to clean and fix up this bug and just combine the commits to make the change log nicer and with that we have this sort of simple pipeline workflow with a simple image promotion um process so in a future video what i want to do is actually do three uh stages right or three different environments of dev environment which is probably unstable because every time the main branch gets pushed to then it would just um yeah redeploy um a staging environment which is a bit more stable and that's what you would use for uh pre-pod right before you actually then say i'm gonna go into production um and both staging and production would obviously grab images from previous environments or build runs actually before that i'm gonna do a series of videos on other infrastructure as codes right um so paloomi and i need to try bicep um and so that is actually those two are going to be the videos i'm hoping to record next week which i can do during work hours which i'll explain then but um probably do that first and then do the cluster so without further ado i'm hoping i can finally publish this video and then you guys have something to watch um this week and hopefully it's semi-entertaining [Music] you

Channel: Julie Ng

Views: 28,255

Keywords:

Id: e0bF1LlclEs

Length: 22min 8sec (1328 seconds)

Published: Sun Apr 18 2021