Setup APIOps for API Management Using Azure DevOps Pipelines

hi everyone in this video we are going to see hanker we set up the pipelines for our API management but before we get started I want to walk you through this diagram which is summarizing what we are going to do in the next few videos first of all we have here our API management instances and this represents different environments like we are going to have three API management instances one for Dev one for QA and one for products then we are going to build some pipelines in azure devops so the extractor pipeline is going to pull API definitions policy information and diagnostic settings Etc from Dev in instance of our API management then having all of these information placed in I get people the extractor pipeline is going to create a PR for us once been reviewed and approved it's going to be merged to the main branch which is going to kick off the publisher pipeline which is going to deploy there's a change into the QA environment of our API management and so on for the prod environment this is at a very high level what we will be doing in the next few videos but before we get into the implementation let's talk a little bit more about API Ops Concepts from where it came from API Ops is a similar concept like git Ops or div Ops where you are combining two practices together or two methodologies together and the way API Ops is working is going to place the API management infrastructure definitions under Version Control so when you make a changes to your API management instance these changes are going to be extracted as code to be reviewed and audited and once have been approved it's going to be deployed to the higher environments like UA and products now let's get started and let's create our API management instances that we are going to use in the API Ops now let's get back to Azure portal and I'm going to create three API management instances I'm going to create a new Resource Group I'm gonna call it RG CI CD Dev and I'm going to put my API Management in Australia east region and resource name is going to be apim CI CD Dev then I'm going to put my organization name and my email address as you can see here the name is not unique so I'm going to put my initials at the end and it should be fine now let's go ahead and create our API management for Dev environments let's go ahead and create it now let's go ahead and create two other API management instances be aware of the cost here guys as you already know each API management instance is going to cost you around 70 dollars a month and I'm now creating three for this section so if you are not prepared to pay for this amount just don't create these instances for CI CD pipelines having said that I'm not going to leave these three instances for a month but just wanted to call it out now let's go ahead and create another API management instance for our QA environment let's create a new Resource Group let's call it RG cicd QA and let's put it in Australia east region and the resources name is going to be API MCI cdqa and my initials events now the organization name is going to be none then I'm going to put my email address and let's go ahead and create our second instance our QA instance now let's go ahead and create our prod instance let's create a new Resource Group let's call it rgci CD products and let's put our prod instance in Australia east region and it's called our API Management in prod apim CI CD prod and then my initials at the end the organ name is going to be none then I'm going to put my email address now let's go ahead and create the prod instance and as you already know in from previous videos it's going to take around 30 to 45 minutes for these three instances to be ready so what I'm going to do now is to go to Azure devops and set up our repos and pipelines until our API management instance are ready for us so I'm going to open a new tab and go to azure devops and I'm gonna go ahead and create a new organization I'm going to call my devops org I've seen Awards apim course and I'm going to host it in Australia east region and let's go ahead with that all right I'm going to create a new project and I'm gonna call it API M and I'm going to keep it private projects let's go ahead and create our projects now let's get back to the documentation there is a GitHub repo for the API Ops solution and if you click on that you should be able to see different tools yaml files that we will be using when we build our pipelines in azure devops so what I'm gonna do now is to go to tags and go to version 4.1.3 which is the latest version now I want you to go ahead and download azure devops.zip all right and let's extract this folder and see what's inside as you can see here there is a tools folder pipelines and delete three EML files these yaml files are simply are going to be used for extractor Pipeline and publisher pipeline which are these two pipelines that we are going to build now let's go back to our azure devops and let's go to repo and initialize our API M repo of course now let's go ahead and create a new folder for tools then Pipelines and I'm going to add readme file and let's go ahead and create these folders let's commit this change now in pipelines folder I am going to upload these three EML files that we have just downloaded tractor publisher publisher with environments let's go ahead and upload these three files now I want you to go ahead and create two more EML files and if you go to the GitHub repo here if we get back to API Ops you should be able to see at the bottom here configuration.prod.yaml so we are going to create two more files at the root here the first file is going to be the profile configuration.prod.yaml and let's commit this change and the second file is going to be configuration.qa dot EML [Music] simply these two configuration files we have just created we are going to specify certain configuration that are specific for different environments as we are going to see in more details later and if you head back to the GitHub repo and click on the configuration.prod.yaml which is a sample file that we will need to put in our configuration file so I'm going to copy this over to configuration.qa.tml let's edit this file first and actually I'm going to clean everything up just leaving the main tags which are apis Bacon's Diagnostics loggers and name the values gonna just keep it very simple but in here I'm going to put the API management instance name for my environment and since I am in configuration.qa.tml I'm going to put the API management instance for qoa environment so let's go ahead and copy the name of our qoa API management let's go back and put it in our configuration.qa.tml file and let's Commit This a change and let's copy this file and paste it in configuration.prod.tml however I need to change the API name to be the name of my prod instance so let's copy the name of my prod API management and paste it in configuration.prod.tml don't worry about these files for now we are going to cover it in much more details later and let's commit this change now what we need to do is to create a service connection so let's head to the project settings and let's go to service connections and let's create a new service connection and let's make our connection type to be Azure resource manager and let's use surface principle and let's go to the next step and let's choose the subscription where you have your API management instance created and let's select the Dev Resource Group that we have created and let's call the service connection apim connection and let's say check on this checkbox to Grant access to all Pipelines and let's go ahead and Save power service connection all right now let's go back to pipelines environments and I'm going to create two environments one for QA and I'm going to add myself as the approver in this environment and let's add a prod environment as well and I'm going to add myself as approver of this environment all right now let's go to library and let's create a variable group but this variable group has to be in a specific name and to know what it is let's head back to our repo and see how we should be calling this variable group if you expand on tools pipelines and extractor you should be able to see under variables section here the group name that we should be using in the library if you want to change it you need to change it here first in the Run extractor.tml file and whatever you are going to put in the extractor file you should be using in the variable group so I'm going to keep it the same API Dash automation let's copy it over to our variable group now we need to add some variables in our group and if you are going to hit back to the repo documentation and I'm going to put this link in the resources of this video and if we go to subtract API artifacts not this one probably in dark configure API tools go to configure API Tools in Azure devops if you scroll down you should be able to see these variable names that we should be adding in our variable group so I'm gonna copy it over service connection underscore name and then I'm going to put the service connection I'm just created apim connection then I'm going to copy API name for Dev environment and then I'm gonna go and copy the name of my Dev API m and I will need to do the same for other environments I need to put apim underscore name underscore QA for qoa environment and let's go ahead and copy the name of QA API m and let's put it here let's add one more variable for prod API m but it's going to be apim underscore name underscore prods and let's copy the name of our apim prod environment all right now we need to provide Resource Group names for these three environments so let's put it for Dev environment first let's copy the resource Group name from here and put it in this variable and I need to do this two more times for qoa environment and for prod environments so let's go to our QA API M and let's copy the resource Group name and let's put it in the QA Resource Group variable and let's do the same with prod let's copy Resource Group name for prod apim and let's put it in Resource Group name prod variable now the last thing we need to specify here is the API Ops version let's copy this variable and put it here and the version that we are going to be using if you head back to azure devops and go to tags this is the version that we have downloaded so this is the version we will be using and configure in our in our variable group so I'm going to put V 4.1.3 and let's go ahead and Save our changes now we have all set let's go ahead and create our pipeline let's create the extractor pipeline so our code is in Azure repo and let's select our apim repo and select the existing Azure pipelines EML files and let's select the extra tractor EML file and let's go ahead with that and let's save our pipeline and let's go to pipelines and let's rename and let's rename our pipeline let's call it extractor and let's save our pipeline and now let's get into the extractor Pipeline and let's run our pipeline now I need to put the API management instance name for the dev environment so let's get back and copy the name of my apim in Dev environment and I need to put the resource Group as well for the API M Dev and here I need to specify the Azure repo which I'm going to be using for creating pull requests I'm going to use API M repo that we have just created and I need to specify the folder where the artifacts will set I'm gonna use artifacts and our Target branch is going to stay Main and we are going to extract all let's go ahead and run our extractor pipeline I need to get into the stage and view this request and permit the access to the variable group however we are going to get an error exactly because I haven't enabled the parallelism in our organization so I need to go to my Azure devops org and configure or purchase some parallel jobs if we go to the Azure devops organization and click on organization settings then scroll down here until you see parallel jobs now let's go ahead and change the Microsoft hosted and self-hosted values let's set up some billing first with my Azure subscription and let's go ahead with that now I need to put some pay it parallel jobs I'm gonna put 25 for both and let's go ahead and save our changes it's going to take few minutes for this change to reflect and so I'm going to pause my video Until the API management instances are created as well so we will get back on run our extractor pipeline again all right now my API in instances had been created for me so let's go back and run our extractor pipeline one more time let's go ahead and run it all right now my extractor pipeline is working all right the first stage has been completed but the second stage is failed while try to create a pull request and let's see what's wrong here as you can see here we need to provide the generic contribute permission on the GitHub repo so let's go to our repos and let's click on manage repos let's click on security let's select apim build service and let's allow the contribution to GitHub repos there are two more settings that we will need to allow but I am gonna keep it until it Flags some error in the pipeline because I want to show you that we are providing the least amount of permissions required in the Pipelines so now we have allowed the generic contribute in our GitHub repo so let's go back and run the second stage and the second stage failed again so let's get in there and see what's wrong and see here we need to allow permission to create Branch so let's get back and allow create Brothers for apim build service and let's get back and run our stage one more time and we are going to see the last error all right and the second stage field again because of a missing pull request contribute permission so let's get back and this is the last thing we need to allow so we need to allow contribute contribute to pull requests and create Branch permissions for API and build service and let's go back and run the second stage one more time all right we have one more error in our pull requests and as you can see here it says there is another repo is pushing to the same reference we are getting this error probably because we have been running or re-running these attempts for this stage multiple times so what I'm going to do is to go ahead and to my pipelines and go to extractor and make a new run for my pipeline so I'm going to copy the name of my Dev API management and then I'm gonna copy the resource Group name of my Dev API management and I'm gonna specify API m and then RT facts and let's go ahead and run a fresh run for our pipeline alright the two stages of the pipeline has been completed now I want you to notice something in the output of the first stage we have rt facts and this is what's been exported from Dev API management so when we run our extractor pipeline the extractor pulls out all of these information from Dev API management then the extractor goes ahead and create a new pull request with this a change requesting to merge all of these artifacts into the main branch so these are simply the files that we have just exported and we are ready to be added to our repo but before I'm gonna go ahead and approve and merge these changes to the main branch I need to set up the publisher pipeline first so let's go back to pipelines and let's add a new pipeline let's go first and have a look at the publisher yaml files so let's go to our repo tools pipelines and run publisher.yaml so this version that you have downloaded from GitHub has only two stages which is pushing the changes to a Dev environment and pushing changes to prod environment and as you might expect it we need to add one more stage to allow us to push our changes to QA environments so I'm gonna go ahead and edit this file and copy the whole block of the posture changes to prod stage and I'm gonna add it here and I'm gonna just change the name instead of prog I'm going to make it QA so these are the changes I've done I replaced all prod names into a QA in that UAE stage and I'm going to leave a new version of publisher file with this change with the new stage in the resources of this video as well so let's go ahead and commit our changes now let's go back into our pipeline creation and let's select our repo existing yaml files and let's choose run publisher.yaml and let's go ahead with that as you can see here our changes has been reflected and let's go ahead and save our pipeline and let's go back to pipeline select all and let's rename our apim Pipeline and let's call it publisher and let's see if our changes now let's get back to my pull request and I'm going to go ahead and approve it and let's complete it by merging it to the main branch and as soon it's going to be merged the publisher pipeline is going to be automatically kicked off to deploy our changes to all environments Dev QA and prod in three separate stages and the publisher pipeline is being kicked off however we are getting some errors so let's get inside and see what's wrong as you can see here we need to install this tool from Visual Studio Marketplace so I'm gonna go ahead and open Visual Studio Marketplace so we can install replace token as it's been highlighted here we need to install replace tokens tool so this is going to be our tool or if you want to search for replace tokens it's going to be the one that we need to install make sure that you are choosing Azure devops on the top here now let's select replace tokens and let's get it and install it in our Azure devops environment let's install it into my org or write all set let's get back to my Pipeline and let's run it one more time and these are the three stages that I've talked about even though we have extracted these artifacts from Dev environments we are going to push our changes to the environment first and this is not a redundant work we want to make sure that all of the environments stay the same at all times and it's not enough to extract the artifacts from Dev API management and push it straight to QA and products maybe in an unlikely event there are some odd things happened in a div environment which is going to leave our Dev environments different than QA and prod that's why after we have extracted the artifacts from Dev API management we are going to push these artifacts into the same Dev API management to ensure that all the environments had been deployed from the artifacts and I need to get in here and approve or provide the permission to access apim automation variable group all right my Dev deployment has been completed so and as you can see here prior to QA and prod deployments I will need to approve the deployment to these environments if you remember I added myself already as an approver on these two environments when we were creating environments under Pipelines so now we are deploying apim in QA environments but as you can see here we are getting some errors so let's get into the QA stage and see what's wrong and if you scroll up a little bit you will be able to see that there is something is wrong here there is an authorization field error and there is a guild here that doesn't have authorization to perform the action on QA API m so let's copy this guide and see what is it let's copy this guide and go to Azure portal and let's search for this guide probably we won't find any results so let's go to Azure active directory and let's search for this good and as you can see here this is an Enterprise application that's been created for me when I created my devops organization to allow the access between my devops org and my Azure subscription and the error that we have got says that this user or this Enterprise app doesn't have permission on this resource so we are gonna go back to our subscriptions and allow this Enterprise app to have an access to our subscription so I'm gonna browse to my subscription go to access control and then I'm gonna add role assignments I'm gonna choose contributor go to the next step and let's select the members and then let's put my name or if you want to go back to the azure active directory and go to your Enterprise app you can copy the name of your app and put it in the subscription here to make sure that you are selecting the right app and now let's go ahead and select it and then let's assign this role all right now let's go back to our Pipeline and let's rerun the QA deployment probably I will need to approve it one more time yes let's review and approve now we have kicked off our QA deployment all right my QA deployment has been completed I need to review and approve my prod deployments let's go ahead and allow my prod environment to access my variable group and let's go ahead and approve the deployments all right now the product deployment has been completed and we have all green for the three stages of our deployment Dev QA amprot and that's not all this is just a very simple setup for Azure pipelines for our API management and in the next few videos you are going to see more complex scenarios in using these Pipelines that's all I have for you in this video thanks for watching and I will see you in the next video

Channel: Hussein Awad

Views: 6,636

Keywords:

Id: 8ZIt_DlNCoo

Length: 33min 48sec (2028 seconds)

Published: Mon Aug 07 2023