Azure Synapse Analytics | Continuous Integration and Continuous Delivery (CI/CD)

[Music] with ever-changing business dynamics and ever evolving technical landscape for it teams being asylum nimble is no longer a choice but a necessity welcome to ask trailblazers i'm arshad ali and in this video i'm going to talk in detail about azure synapses native integration with azure devops and github i will then demonstrate how we can set up git repository and build pipelines for continuous integration and delivery of synapse workspaces artifacts from one environment to other environment continuous integration or in short ci is the process of automating the build and testing of code every time a developer commits a change to a version control like git repository in a consistent and automated way whereas on the other hand continuous delivery or cd is the process of building testing configuring and deploying a unit of build from one environment to another environment for example promoting a build from development to test and from test to production cicd bridges the gap between development and operations team by automating the build building testing and deployment of application in synapse promoting a workspace from one environment to another environment is a two-part process first by publishing feature we create an azure resource manager template or insert arm template which acts as a deployment unit then we deploy this deployment unit which contains artifacts like sql scripts notebooks spark jobs definition pipelines data set data flows by using azure synapse cicd tools in azure devops or on github in this demo i have these three workspaces created and with each of these workspaces you'll see there is a suffix which denotes the environment that it belongs to right so in the first case uh i have tape and the second one i have uh and prod this is not a necessity you don't need to name your workspaces like this i have named it in this way to make it simpler to understand we are moving from one environment to other environment you have full control in how you name these workspaces and you still get the benefit of moving workspaces artifacts from one environment to other environment with that let's look at the day of life cycle with azure synapse the very first thing that you notice over here is we need to do is to create github repository projects or we have to create a projects in azure devops and the first thing that we have to do right after that one is to create a collaboration branch we often name it as a master branch then if a developer starts working or a team of developer start working on any feature we need to create a feature branch for them to kind of collaborate and build the solution or the feature in that branch they'll keep on committing their changes into a working branch for that specific feature and once they are satisfied with their changes they'll create a pull request and that will trigger a email to email or notification to a reviewer to review the changes for that feature and approve it once this gets approved the changes from working branch or the feature branch are merged into collaboration branch or master branch at this time if all the features have been built and it's ready to be deployed into the next environment you can publish those changes from master branch to publish branch and this is where the arm template gets created we look at those details once we get into the demo but this is where the deployment unit gets created and stored at this time you can take this arm template and create release definition in azure devops you can create a release pipeline and you can define how this is going to be deployed you can map all the parameters from one environment to other environment and then once the release definition is done you can either manually deploy that release to other environment or you can set up the automation to be deployed into the next environment so this release gets deployed to the test environment and once all the test cases have been passed you are all set to do it to do the deployment into the production environment so in a nutshell this is overall devops life cycle for majority of the azure synapse application that you will be building with that it's time to look at how to set it up in practical scenario as a prerequisite we first need to make sure we have azure devops organization already created and then the next step is to create the project in that organization in my case i have this organization already created so i'm going to create a new project so that i can use it with my user synapse workspace you might have a project already which you can leverage to connect it from synapse workspace but in my case i'm going to start it from scratch so let me click on new project and create a new project for myself so i'm going to give a name here and then i need to specify the description that i want to provide for this project here it goes and then in terms of visibility of this project what level of visible visibility i'm going to provide right whether it's going to be a public devops repository or enterprise level or it's going to be private in my case i'm going to make it private because i'm only working on this one for now so let me go ahead and select private and then click on create to create this new project it's going to take couple of seconds and once the project is ready we can have a look so here we have uh now our first project created our azure devops project created if we can look at the repository uh we'll see that there is no repository it created it's planned for now there is no branches is created it's planned for now now as a next step we have to install azure synapse workspace deployment extension in this organization that i'm using right and to do that let me go to the home page click by clicking on azure devops this will take me to the organization i'm part of and on the bottom you'll see an option to change the organizational setting click on this it will take us to a new screen where we can see there are multiple options here on the left side and i want to install an extension for synapse workspace deployment click on this for now you'll see i don't have this deployed yet so i need to kind of click on this browse marketplace specify the name of the extension that i want to deploy right in my case it's going to be synapse workspace deployment right click on search this is what i need to deploy for ci cd operation click on this click on get it free it's asking for the organization where i want to install this extension in my case it's a ar ali data click on that and now my um extension has been created or are installed in my organization if i go back to my extension page and refresh it one more time i will see that extension getting listed over here at the bottom right so this extension is installed here once we have met the prerequisites from azure devops site we can go to um azure synapse workspaces that we have created right to do that let me go here and this is my resource group and inside this resource group if you notice there are three workspaces created one for day prod and ui then each of these workspaces has default storage location or basically a data like a storage account associated to it and because i'm going to demonstrate creating a notebook which will run on a spark pole i have created a spark pool in each of this environment so if if you have a notebook which references a spa pool it's important to create those spa pool in all the environment where you are going to do the deployment or else the deployment will fail and i'm going to demonstrate that so if you see here i have created this spark pool in my dev and uh environment i have not done it for production and i'll show you uh when we do the deployment in production it will fail and we and and before we can make it work we have to first come here and create a spark pool for the production right so for now let's keep it simple let's start with these three workspaces and as you can see these three workspaces are in a single resource group and they are all part of same subscription but in your scenario it could be in a different resource group it it could be in a different subscription and it's all fine the the deployment process is not going to change i'm going to be perfectly fine the way we are going to do the deployment now in my example i have created three workspaces dave uh and broad but you might have more than three right so you might have dave to test you had to plot right it doesn't matter how many environment that you are going to create the deployment process is going to be same you just have to kind of follow the same process and you need to do it for as many as environment you want to kind of do the deployment on and we'll we'll look at those those deployment pipeline so once we have this prerequisite of creating this workspaces are done we can go to synapses studio for each of these workspaces and have a quick look so this is my dev workspace and this is a synapses studio where i have opened this workspace and if you come here and open uh click on develop tab on the left you'll see that this workspace is currently connected to synapse live so synapse live is where where like all the artifacts that you are going to create in this workspace they are going to be stored as part of the synapse service but this is what we want to change we want to change it to store all these artifacts in git repository that we just created and to do that we have to uh either click on this place on this combo and click on setup code repository or we can go to manage and on the on the tab list that we have over here go to source control heading and then under that heading we have git repo git configuration click on this click on configure and then specify the repository type that we want to use azure synapse supports azure devops as a git repository or you can use github as a git repository it could be enterprise github or it could be public github i'm going to use azure devops uh for now so let me select this as your dev devops and then it's going to ask i'm going to ask me the azure active directory tenant where i have this devops organization and project created so i select that and if i click on continue it will take me to the next page where it will ask me to choose the organization that i want to use for connecting this workspace to right so in my case it's a r ali data i'm going to select that and it's going to show me all the projects on in that organization so i'm going to use the project chat that we just created synapse cicdemo ado right and then we want to create the um repository so in this case i'm going to give the same name for the repository as the project name right and then in in in terms of collaboration branch we can specify the name of the collaboration branch it's it's recommended to create it as a master that's a standard that people have been using so let's go ahead and let's create a master branch for collaboration so my club collaboration branch is going to be master publish branch is going to be by default named as workspace underscore publish let's keep it that way so let me click on apply and it's going to take few seconds before it can create or connect to the git repository at this time as you can see the configuration is done and we have successfully able to connect this workspace to the git repository that we created right in case uh in in um in case if you want to kind of disconnect it from the git repository and in order to use it in a synapse live mode there is an option here to disconnect it from the git repository for now we don't need to do that so let's go back again to the develop page or develop tab here and if you see here on top of this combo master branch is selected right now suppose there there are a couple of developers who are going to be working on a new feature so for them we need to kind of create a new feature development branch right so let's say uh we want to do that and for that we need to create that branch so click on this new branch options over here and then specify the name for for that branch and you can give any name that makes more sense for that specific feature i'm just saying new feature in this case and i'm going to create this branch based on what is already there in the master branch right so let's click on create and as you can see we have this branch created if we if you cannot click on this combo you'll see this is the master branch this is the branch that we just created and because we have not created any artifact so far you'll see nothing coming over here right and this is something that we can validate by going to the uh azure devops repository that we have right so let me go back there let me go here and this is a project that we created and if we look at the project that we create here go to the repository at this time we'll see there is one repository created by the same name as the project right so this is the name of the project and this is the name of the repository as well and if we go to the branches we'll be able to see that there are two branches one is the master branch or basically the collaboration branch we named it as a master and the other one is new branch that we created for uh the feature that we are going to develop right and at this time uh if you look at uh if you go to any of these branches because there is nothing created yet you will not be able to see anything coming over here so we have credential integration runtime and linked services uh but um there is nothing created these are all by default that comes in as a artifacts so let's let's go ahead um let's let's go back to the synapses studio for development environment again and at this time let's create a notebook right so to do that click on go again to the develop tab if you have if you are not there already click on this develop tab and on the develop screen over here click on the plus sign and click on notebook and then specify the name of the notebook that you are creating so i'm going to name this by this name i'm going to say synapse ci cd demo notebook right and then you can put the command that you want to execute as part of this notebook for simplicity sake what i'm going to do is going to simply read for simplicity sake i'm going to read the data from public open data set and write it into the default storage uh location that we have right this is a very simple command but you can have as complex as command that you want to write into your notebook the process is going to be same right and because this is a notebook this notebook gets executed on a spark cluster we need to attach it to any of the existing sparkful that we have created in this workspace i have this one sparkle created it's named spark pool version 3.1 i'm going to select this and and at this time um i'm going to keep it simple you can have as many as uh sale you want to add to this notebook and as as as complex as processing that you want to do but for now let's keep it simple uh at this time i can click on commit when i say commit this is being committed to the feature branch that we just created right there is an option to create click on commit here this is going to be kind of committing the changes into this notebook if you have multiple notebook and you have messages into multiple notebook you can click on comment all where it will kind of do a batch update to all the artifacts in one go right uh clicking on this commit is going to be kind of committing the changes only specific to this tab that you see over here and in this case it's a notebook so let me kind of go ahead and click on commit my notebook has been submitted and if i go back again to the repository that i have at this time if i go to branches and look at the different branches that we have and you can see this my new feature branch is one change ahead of my master branch and this is the one change that we just made right if we click on this new feature brand you'll be able to see that notebook folder now appearing over here and the notebook that we had there is coming over here right so this is this is all good we can see the changes is coming into the feature branch that we have created so this changes are still in feature branch this has not gone to master branch and we can go and validate that right so if we click click on branches again and go to master branch we see that the changes was only committed to feature branch so we do not see a notebook folder here or the notebook artifact over here yet right and we'll see how to move that into master branch so once we have this notebook created or are kind of created and committed to the feature branch let's go ahead and create some other objects so let's go ahead and create a pipeline right so in this case i'm going to create a pipeline to call this notebook or execute this notebook right so click on integrate and click on plus sign under integrate this will give you an option to create a pipeline click on this pipeline and let's name this pipeline right and what i'm going to do in this pipe will line is i'm going to add a activity to call the notebook that i just created so let me kind of hide this for a moment just to get some more space and let's say we want to name this activity as nb iphone call or basically we are calling a notebook click on settings and this is going to give us all the notebook that we have in this workspace we created this notebook so we are going to simply say this is the notebook that i want to execute right and all the errors have gone now and we want to commit these changes again to the feature branch that we have created so let me kind of hit on comment and then you'll see this artifact has also been kind of committed to the feature branch that we have and again we can go to branches here we can look at a new feature branch and you see this this feature new feature branch is two changes ahead of master branch and we can go to this branch and we can look at this pipeline folder and this is a pipeline that we just created and we can see it's coming over here assuming that this is one of the feature that we just developed and uh we are good to go ahead and do the deployment of this changes from development to test uh in actual scenario you might have multiple features being developed at one time and getting deployed at one time but to keep it simple let's say we we just have created a notebook we just have created a pipeline and that pipeline is calling that notebook right and this is one feature that we want to go go ahead and do the deployment of right so the first thing that we have to do here in this case is we first have to move the changes that we have in the feature branch to the master plan and in order to do that we need to create a pull request so developer has to create a pull request um we can do that here in azure devops portal or we can do that in synapses studio workspace so if we go back here if we look at the combo again here we'll see there is an option to create a new pull request right so click on this create pull request and it's going to take us back to azure daybox again and it's going to ask us to kind of provide a title for the pull request that we are going to create and let's say this is something that we want to specify over here so we are going to kind of most new development feature notebook and pipeline to the master you can provide a description as you see fit and then you can say create right so at this time pull request has been created for reviewer to kind of review the changes and approve if it looks good to take it to the next step or a reviewer can reject it so in in that case it will go back to the developers to work on it to fix those review comment or incorporate those review comments considering that this feature is complete and i am the reviewer i can go ahead and click on complete the moment i click on complete it's going to ask me if i'm ready to do the complete merge from feature branch to the master branch so once the merge is complete during that merge it's going to take all the artifacts that's there in the feature branch going to merge it with the with the artifacts in the master branch and delete the feature branch so let me go ahead and do that let me click on this complete merge and it's going to kind of run for few seconds while it's doing the merge and now we can see this merge is complete if we go back again to the branches you'll see that there is only one branch now master on the feature branch that we had earlier where we made the changes or committed the changes those changes have been now merged to master and master is most up to date with all the changes if we can click on master we'll see that the notebook and the pipeline that we created are part of this this master branch at this time now if you go back again to synapses studio portal and refresh the page you'll see the same status being reflected over here so let me refresh the page once it loads uh you can again click on this combo and you'll see this we just have master branch we don't have any um kind of feature branch as of now right so at this time our changes have been merged into the master branch we can continue to let developers work build a feature and start merging those features back into master branch and once the the the number of features which are supposed to go into the next environment is ready to be deployed we can click on publish so when we click on publish what it does is basically it takes all the artifacts from the master branch and it's going to create the arm template arm template is basically azure resource manager template uh with all the artifacts embedded into it so basically it's a it's a deployment unit that's going to be deployed into the next environment so when we click on publish branch sorry when we click on publish it's going to create those artifacts for us so let me do that and if you see here it's saying that um these are the two changes or two um artifacts which have been added and this is why it's saying added because we have this new keyword over here if we are making any change it will be updated uh in our case it's uh it's all new artifacts that we have added click on ok and this is going to go ahead and create the arm template for deployment so as you can see um templates have been created and we can go back again to azure devops and if we can click on branches we'll see there is one more branch over here works workspace publish and if you can click on this workspace for place you'll see there are two artifacts created over here template for workspace so this is the deployment unit or this is the arm template that has been created by that process and it includes all the artifacts that we have we had in the master plan so it's a single unit of deployment and you can see this by looking into details of it so we created one pipeline and we named this pipeline um by this name right so this is one component um or artifact that we had in the master uh repository likewise if we go down you'll see there is one for a notebook also right that's that's uh created over here let me go back to the notebook right so you see this there is a notebook also created over here right so all the artifacts that that were there part of the master branch are included in this template for workspace.json file this is the arm template which will be used for deployment from one environment to other environment uh the one thing that you will notice over here is um the name of the workspaces that we have created we have created it differently from one um from one environment to other environment and that was intentional so if you look at the name the dave has dave has their keyword in the in the uh name whereas the uh has you at and then um then then broad has pid so we we have to parameterize this right so we want to take some artifacts from dave we want to do the deployment into uh environment but the uh environment has a different name than than the development environment so we have to parameterize that so there is another file created along with this template for js template for workspace dot json file and this is where you'll see the those those parameters are defined so in our case we have one parameter called workspace name and this is the workspace name for now because this is coming from tape but as we know we are going to do the deployment in ui so dave has to be reply replaced by uh so we have to parameterize it right and this is the one file that we use to do the that parameterization and this time since we have published the artifacts or which we have we have created the arm templates for deployment uh we can go to the pipeline tab on the left of of azure devops page and we can go to library click on library we want to do the parameterization of the variables right and then click on um plus variable group so we are going to create two sets of variables one set of variable is going to be used for ui deployment and other set is going to be used for production deployment in our case we just have one variable in each of the environment but in real deployment you might have um more than one variable right so you can combine them together in a single group so let's go ahead and create a variable group for our uh environment to do that uh let me specify a name over here and this is the name that i'm going to give for the variable group for you at and i'm going to add one variable here and i'm going to name it as a env and this is going to be the value of you at right so this is one variable group with just one variable in in your scenario you might have multiple variables to be used uh during that deployment to that specific environment right with that let me go ahead and click on save right i have to do this process for a production environment also right i can go back to library again and click on plus variable group one more time to create one more variable group i'm going to keep a different name this time for broad and in this case it's going to be hyphen prod at the end and again um this is a variable group and in this we have to create a variable right and the variable is going to be environment and this is going to be crotch prd right and then with that i'm done with creating the variables to map it or parameterize the environment name if i can go back to library tab again i'll be able to see uh both this variables group have been created with the variables one variable in each right so this is done once we have this variable groups created we can go ahead and create the release pipeline right so this release pipeline is going to take the deployment template or deployment unit that has been created by publishing uh from master branch and use these variables to map the parameter and do the deployment on the target right so let's go ahead and click on release tab on the left on this azure devops portal and click on new pipeline that we want to create and we want to create it based on an empty job right so let me click on empty job here and then i want to name this as uf right so basically what i'm going to do is take the published artifacts from uh the dev environment and ah and and deploy it to the event environment so this is this is this stage is going to do so let me close this and i'll explain you one more time so if you see here there are two kind of um component over here the one is the artifact that we are going to take and do the deployment on so this this one is the ui deployment uh stage where we are going to do the deployment on ui once uh stage has been created we can go to variable tab over here and create a variable which will map the variable that we just created under uh library tab as part of the variable groups that we have created right and the the idea over here is we have a variable in release pipeline and that will be parameterized based on the variable that's coming from the environment variable that we declared earlier with the specific name value over here so click on add and let's say workspace it's the name of the variable and this is the value for this variable and if you see here this value is going to be dynamic and it's going to be coming from the variable that we declared as part of the variable group earlier right uh and and in that variable for you at we we said the value is ua right so once this is done let's go to the variable group and we can link this variable in the release pipe pipeline with the variable group that we created earlier so in this case it's going to be you add we are doing the deployment to the uh environment first and what's the scope of this variable group right in this case i want to limit this scope to uh environment so we can say uh if i have a variable which is being used across multiple stages i might like to keep it across all those stages by checking release but for now i am going to be using it within this stage so let me uh make it as a uh stage and link this once this is done uh we can go back and we can validate and we can go back again to the pipeline that we just created at this time we have created a you at stage we have created variables to kind of parameterize the variable value for the workspace and now what we can do as an step is specify the source from where this artifacts will come right so for that let's click on this add an artifact link over here this is going to ask for the source from where this artifact is going to come and in this case because our arm templates are deployed or are kind of stored as part of the workspace underscore publish folder we can click on azure repository and this is going to give us the name of the project where uh from where we have to pick it up in my case it's stored here in synapse i1 ci cd hyphen demo iphone ado and this is the project name and repository name is again going to be the same as what the project name is and in the next step i'm going to choose the uh publish branch from where i am going to kind of get those on deployment uni and then i have to specify a name to to be used as an alias to define it as a source right so in this case i can say date so this is my daily environment work space underscore publish from where this artifacts are going to be coming and will be deployed will be deployed into the event environment so when i click on um add once the source for the artifact is defined the next step is to go ahead and add a task for doing the actual deployment to the event environment so click on this link this will open up this screen where we can specify um the task that we are going to add click on agent job and click on plus sign over here to add the task at this time we need to add a task of type synapse workspace deployment this is the task that we are going to do that going to be added to the release pipeline and this task is coming from the extension that we added in the very beginning of this demonstration so click on this one to add to uh this this stage right and this task requires some settings to be specified so let's click on this at this time we need to specify the template the arm template which will be used to create those artifacts on the target right so we we in the in the previous screen we selected the folder where those templates are available at this time we are going to browse through that folder to get the actual name of the template file so in this case it's going to be template for workspace.json file right likewise for parameter we need to select the parameter file so in this case if we go down to the same folder again we'll see template parameters for workspace.json file is there click on that and then um the next one is how this synapse um how azure devops is going to be connected to uh the synapse workspace right and for that like let's go ahead and let me select the um subscriptions that i have and authorize azure devops to connect to my subscription and to do the deployment into um resource group so let me click on authorize for now this is going to take couple of seconds so once the authorization is successful um this will show you the message like this so what we have done so far is basically we have authorized azure devops service principle to connect to the subscriptions that we have specified and and do the deployment in that uh subscription for the given uh workspace that we have synapse workspace that we have so in this case i'm going to be using the resource group that i have where the subscription exists and then i have to specify the name of the workspace where the deployment is going to be done right so in this case this is my event environment so i'm going to be specifying the name of the uh environment right and at this time we also need to overwrite the parameters right if you remember from our discussion um we have a parameter in templates per parameter dot json file and then we have one variable that we have created and we have another variable that we have created in the release pipeline so this release pipeline variable is going to get the value from the variable group that we have created and we have to overwrite that to write it to the um into the during the deployment process right and to do that we have to say workspace name this is the name of the variable in the template file and we are going to use the variable name that we created as part of the release pipeline right so we are going we specified it as a workspace so we'll say dollar and then in the bracket it's going to be workspace click on ok and then click on save this is going to save our changes so at this time uh what we have done so far is like if you go back to the azure day box process that we discussed earlier or the life cycle that we discussed earlier we published the artifacts from master to the public's folder and then we created the release definition so this is what that release definition is so we have source we have target then we have the mapping of the variables and then we have task to do the deployment um to the to the target environment now as part of the delivery or as part of the deployment we have to go ahead and create a release so click on this create release here and this is going to be kind of triggered manually right so let's go ahead and click on create and let it run we can click on view releases to look at all the releases that has been created from this pipeline and this is the release one which is uh which is something we just have uh kicked up if you can click on this one it's going to show you the progress as as it starts deploying to the target environment so i'm expecting this to fail once this fails i'll tell you why it failed and how to fix that right so as you can see this has already failed we can click on this log and go to the details screen where you'll be seeing all the steps that gets executed in order to do deployment on the target side so this is the exception message that we see over here so if you click on this exception message it takes you to a screen where it has the detailed message about why it failed to fail to deploy it into the target environment right if you look at the message this is what it says it says that the principle this the the id that that is given over here doesn't have required synapse our back permission to perform this action so it means that even though we have given permission to kind of connect to the target subscription where this workspace exists the service principle which is being used by azure devops to do deployment into that workspace doesn't have appropriate permission on azure synapse right so just copy the subscript uh um the print service service principle id from here and let's go to the event uh synapses studio that we have right in this case we can go to manage this is my event environment or like synapse workspace for you at environment and if we go to access control we can provide the right access to the service principle which is being used to do the deployment so in this case come here and click on add select the role that we want to provide and in this case the permission is needed to do the deployment on in this workspace is synapse artifact publisher so select synapse artifact publisher over here and then just paste the service principle id that we had from the error message right if you do this click on apply and this gets at this time as you can see um the service principle which is being used by azure devops has been added on target synapse workspace with appropriate permission to allow it to do the deployment of the artifacts so this is something that we did it in the event environment and because i'm going to do the deployment into production environment also right so let me repeat this process for the production environment before we can do the next step right so to do that i am into this synapses studio again but this time for the production environment and if i go to manage tab on the left go to access control click on add over here and then select the role that we selected earlier synapse artifact publisher specify the same service principle id because the same service principle id will be used to deploy in both the environment right so click on apply and with that we are done with giving all the permission for it to kind of do the deployment on right at this time we can go back to the release pipeline that we had earlier and close this error message and we can say deploy it again so at this time because we have given right privilege it should be able to do the deployment right so we can click on this and it's going to be queued and going to be doing the deployment very soon at this time you can see that the deployment to uh has completed successfully so if we go to the uh pipeline again over here you see this the deployment to you it is succeeded successfully and we can validate this by going to the synapses studio for um for for the ui workspace that we have right so click on home and let's go to develop and we should be able to see a notebook that we have deployed it through the deployment process let me refresh the page as you can see we have this notebook deployed over here right so we can click on this and we can see this notebook is here deployed into uh environment if we go to integrate and if we look at the pipelines you will see the same pipeline is also deployed here in this workspace right now the one thing that we have to keep in mind is when we are integrating or connecting a workspace with azure devops repository we do it only for the development environment where we want to maintain the version control or maintain the history of changes that we are doing uh and and from where we are doing the deployment to the next environment or to the upper environment like test or uh or production in the workspaces for test or uh or production these are all going to be connected to synapse life right so that is very important you would not be connecting test or ui or production environment to get repository the only development environment will get connected rest other environment will be connected to synapse live and these environments will get the artifacts deployed as part of the release cycle through the ci cd process that we have built into uh the azure devops so that is very important to consider now let's go back to the release pipeline that we created earlier right so this is the pipeline that we have created and we have done the deployment to the event environment so this is the status of the release one that we just completed right uh what we are going to do next is um going to change this release pipeline or edit this release point to pipeline to include production as well right so this is the release the execution instance of the release pipeline in order to um edit the pipeline is itself we have to click on a date and click on edit pipeline this will take us back to the pipeline that we created earlier right so in this since we have deployment to you it completed successfully we can go uh next step and we can add deployment to production as well to do that let's go ahead and clone the stage that we have for your deployment the process is going to be very similar to do the deployment in um production environment however there will be some changes that we will have to do but before that let's go ahead and clone this right so the very first thing that we need to do here is make sure we are naming it appropriately so this is going to be deployed into the production environment so we have named it prd right and the next thing is we have to kind of click on task and make sure we have this task pointing to the right environment so if you can see here we have synapse workspace name and we have this name dynamically being built based on the based on the value of this variable and this variable is what we have defined for this different environment right so if we go here this is coming from the variables that we have declared earlier now in this case uh because we have this uh kind of a workspace created for different environments so we'll go ahead and we'll add this to uh this variable list again uh this is this was for a uat environment this is for uh environment and we'll add it for um production as well sorry uh it's going to be workspace and then this is going to be production and we'll change the scope this one is going to be you at and this one is going to be raw to differentiate that these two variables are from two different environment and if we go to um variable group uh we can see that um we need to add uh production as well so let's go ahead and add broad and say this is going to be tied to product broad environment right so this way we have linked this right so this one is going to be um uh only so i think we need to edit this uh change the scope we have to say this is going to be you at only and save this so we have to set up variable group one is going to be applicable for you at one is going to be applicable for prod right and then we have two set of variable right and these are all uh from different environment right so this this way we are good here so far um so what we do need to do next is uh i'm going to set up this entire deployment process and i want to make sure every time when there is a change in the publish repository or basically when a new release is being published to the publish uh publish branch i would say in that case there should be automated deployment to the event environment and to do that we can click on this thunderbolt here on the source artifact link and click on enabled right and then we need to make sure we are selecting the branch from where it will be picking up the new changes right so once we have selected this and saved this information it means that every time when a new publish is coming in or basically a new template is being generated by clicking on the publish in the workspace that will trigger the deployment automatic deployment of the bill to the uh environment right so this is something i'll save it here and i'll say i'm i'm done with this deployment into in event environment however for production environment i do not want the deployment to be automatic right i want it to be based on uh if someone reviews it and approval approves it and then it should be deployed into the production environment so basically the point is once the deployment is com development is complete move it to uh environment run all the uh and uh test cases and ensure everything is taken care and once it passes all the tests then approve this to be deployed into production environment to do that click on this pre-deployment condition and then go to pre-deployment approval section over here enable this and then specify the name of the person who is going to be approving in this case i'm going to put my name to make it simple for approval right and um this this is the timeout that we can specify right so this is 30 days by default i'll keep it that default as is and click on cross here and then save this so with this all my changes have been done now i can go ahead and kick up the process of deployment um from from end to end perspective right and this will also be taken care or will kick off when there is a new publish to the publish brand for now let me kind of go ahead and create a release based on what we have so far so i'm doing it manually for now click on create and go ahead and click on create again and this has started the deployment i can click on this link here on release 4 or i can click on view releases and this is where i'll see all my releases if i click on this release this is showing me the current release instance it's going through the deployment so let's wait for a couple of seconds for it to complete and then we'll come back and look at it now as you can see uh deployment to the uh environment has successfully completed and um deployment to production environment is pending for approval at this time i can go ahead and approve this request once uh we have verified that all the test cases have been passed successfully and it's safe to do the deployment in production environment so this is where i can approve it and kind of start a deployment into production environment but before that let me show you something else while you can see there is a pending deployment to production environment in azure devops um because i'm the approver i i received this in this email from azure dev devops intimating me that there is approval pending for me to start you know deployment in the production environment so this is where i can click on view approval to directly jump onto aj devops to do the approval process and i start a deployment or i can go back to this portal and click on approve to start the deployment process click again approve here and this will trigger the deployment to the production environment so let's wait here and see what happens uh i'm expecting it to fail um and we'll look into why it fails and how to fix that in a while [Music] as we were expecting it to fail it as well if we can click on this one it's going to take us to the page where it actually failed we can click on this and this so just error message by what what caused it to fail right so let me go here and show you this error message and if you see this it says that data factory resource provider that component has failed to do the deployment and this typically happens because uh if you remember we have a notebook which gets executed on a spark pool and in my dev environment and in testing environment or uh environment i had that spark pool already created so let me go back to my azure portal and show you this right so if you see here i have one spark pool created in development environment on which i executed my notebook or execute my notebook likewise i have a spark pool created with the same name in event environment where i did the deployment uh through the release pipeline and it was successful however i do not have this but sparkle created on the uh in in the production environment right and that is why it's failing so what we need to do is basically we need to go to um production workspace that we have and create a spark pool with the same name what we have in the development environment so in this case it's a spark pool version 3.1 is something the name so let's go to the production workspace uh synapse studio here and click on manage go to apache spark poll click on new spark pool specify the name here i'm going to give the same name right and i'm going to disable auto scaling for now just to keep it simple let's make it three and let's go to additional setting and say point one is this patches path version i'm going to create and click on create and this is going to create the spark pool right so once this is created we can go back and kick up the release pipeline again and this should be kind of completed this is getting deployed let's wait for a minute so a spark pull deployment is completed in production environment so we have the prerequisite where we wanted to make sure that the spark pool that we had in our development environment is also there in all the um kind of uh deployment all the environment where the deployment is being done so right so we have uh made this prerequisite to have it in the production environment now we can go back here and close this and we can say deploy again and let's go ahead and click on deploy again and let's see what happens and this time yeah it's indeed completed successfully if we go back to these four by clicking here we can see that end-to-end deployment of this release pipeline has completed both to uh environment and production environment as i wrap up this video i would like to highlight these three things first make sure azure devops service principle has synapse artifacts publisher role assigned in the target workspace second integrate only development workspace with git repository and third prepare spark poll before you migrate your artifacts into your target workspaces thank you once again for watching please do like subscribe and let me know your feedback or any specific topic you would like me to cover next looking forward to see you all soon in the next video

Channel: Arshad Ali - Aas Trailblazers

Views: 929

Keywords:

Id: P6PsVQa0Y3M

Length: 64min 0sec (3840 seconds)

Published: Tue Nov 02 2021