Deploying Infrastructure Automatically To The Cloud Using Ansible and Azure Pipelines

hello everyone welcome to our video deploying infrastructure automatically to the cloud using ansible and Azure pipelines in this video we are going to see a complete tutorial for how to meet the creation of the infrastructure using infrastructure as code with ansvil and Azure pipeline so first I will give you an overview about the demo workflow this will give you better idea and abstraction of all the steps in the demo also I will during the workflow I will explain and give more knowledge about the component and the compilation so it is not just the workflow but I will also explain and describe many things on this component and configuration during explaining the work of law then we will go for the demo but first let me give you a quick introduction about myself my name is Mohammed radwan I'm a developer technologies MVP and principal DevOps consultant I have been doing software development for more than 15 years now working on several projects for different enterprise customers across different region and countries i heavily participated in the community developed several frameworks presented mini sessions holding various certificates for azure and devops i started back in 2002 as a classic ESB developer relocated multiple times in multiple countries working with several technologies and for different role and positions and for the last eight years I moved to consultancy based role where I focused more on helping different companies some of them from the fortune 500 to improve their software development and delivery using DevOps and automation I have developed and authored several frameworks command line tools and guides you can find them on github and agile devops marketplace I like to share my experience this is Hawaii I have a YouTube channel and blog where they have hundreds of videos and posts that share different topics from my real experience I have presented mini sessions in several user group conferences events and customer sites as well in different countries so I got the chance and the opportunity to work in different culture project size and company size as well which helped me to have different experience here's some links for my activities ok so we will start the demo workflow on my client machine I have my Windows machine with a kid bash installed so I can use a gate pass as a terminal connection to a remote Linux machine and the next step is to start login to my portal and start creating resource group and start creating a remote Linux machine I need an open two machine because I want to use this machine as a remote and civil machine so I will create this machine I need portal 22 open so I can communicate to this machine with the SSH once I complete creation of the machine I will login to the machine from my local machine using the SSH and they username and pass and then the first step is to start installing and configure as well on this machine so I will prepare this machine to be a remote and simple machine once I complete the configuration and demonstration of ansible and part of that also is to install the azure is the key for ansible and test my virtual machine to make sure that unspell is running correctly and then I will start configuring the authentication from that machine to my Azure subscription I need to create a credential file inside the virtual machine so this virtual machine can has access to my other subscription because it's going to create an infrastructure on my adress subscription so I need to give permission to that virtual machine so I just create credentials file and then after that I will generate an appear RSA keys private and public key after that this machines is ready then I will go for a DevOps and then navigate to the project setting and start using the private key to create a service connection the main idea here I need to make as your pipeline have authentication to the virtual machine so it can communicate with the virtual machine because I will use Azure pipeline to execute and civil playbook remotely on the NC Bell virtual machine so I need to authenticate to have a permission the pipeline to that machine once I complete that I will run a continuous integration build the pipeline and since this is a java application it will store the marvin task which will start doing all the tasks inside the poem or the project object model which is restoring all the dependencies running all the unit tests and so on and then at the end it just creating the java package which is a war file after creating the package then storing the package on Azure DevOps artifacts so it can be ready for the CD or the continuous deployment pipeline also it will store the ansible playbook on the same artifacts so it can also be ready to picked up by the continuous deployment pipeline so what is the NFL playbook to channel so the ansible playbook the Chairman is just the infrastructure as code using the anvil with the ml configuration so this describing all the infrastructure that will be created using the pipeline so it just a metadata that describing the infrastructure the main idea here is that the continuous deployment pipeline needs the ansible playbook so it can create the infrastructure needed to deploy my application and then Bix the package to deploy that package to the created infrastructure so this will be available on the artifacts for the continuous deployment pipeline and then I will just run the continuous deployment pipeline the main idea here first the pipeline will pick up the NCL playbook which is the ML file and then start running that remotely on the remote ansible virtual machine and this execution will start creating the infrastructure described in the ML playbook which will first create an azure service plan and then whip my sequel database and then also configure a firewall rule to allow a sure web app to access my sequel database once the infrastructure is created and ready then the pipeline will start pick up the package and then deploy that package to the web app which is a word file and then extracting all the Java file into the web app so the web app now has all my application deploying and then this application will use the my sequel database then after that I can open the web application from the browser and navigating to the hotel sample application and even log in to that and see all the data that loaded from the application and the database so let's now go for the demo so let's first see the project the demo generator and this is the project that I will use during the demo smart hotel and this is architecture of the application and this is the application which is web application and mobile application as well but we will use only the web so listener we get to the demo generator sign in and then here I will choose the template navigate to develop slab and the choose ansible and select this template then here I will select the desire organization so as we can see I need to add a pipeline to extension to be in stool on the target as a devops organization which is ansvil and replace token so let's open both of them so i can install them on the organization get it for free and here i will select the target organization and click install now it's complete let's install the replace token the same organization and click install now it's complete and now let's refresh that so we can see that now the two extension installed and my organization is ready to create this project template so I will just name the project ansible project and create project so this will create all the artifacts of the project including the work items and build the definition and so on so let's now refresh the other DevOps so we can see the ansible project created let's now navigate to repos here's a project and we can see this is the source the main source and we can see it's an MVC application using Java so let's navigate to the azure resources so I can execute ansible playbook on a remote machine and this machine needs to have installed so it's needed for the desire cloud providers and in our case as I'm going to work with other cloud so I need to have an azure suite on the remote machine so I need to have a remote machine which will be a Linux VM which has an ansible installed and configured as well as other suite for the azure cloud providers so I will start the step by preparing that machine so the first step is to start creating the service principle you can watch this video for more information about docker contra narration and where and how docker can fit with DevOps and CI CD pipelines in order to do that I will open the cloud shell so this will load a cloud chill let's say clear the screen and then I will type this command to create the service principle with the name service principle at one this will generate the service principle and carry the service principle and I will open a new notepad and copy the output in the notepad because I will use it later then type the command as your account Cho which will show the azure subscription information also I will copy this information to the notepad copy and put it in the same not bad and now I'm going to create the information that I will use for the authentication so here I will create these values the subscription ID the client ID secret tenant and then I will copy this information from here so for example the app ID it will be the client ID and the secret is the password and of course the tenant from the Azure subscription so prepare this information then the next step is to create the virtual machine with Linux or 0.0 I can use this link which will create a VM with Linux and ansible install and configure but I prefer to show all the steps from the beginning on how to create the virtual machine and how to install and configure ansible on the machine so I will start that by creating a clean Linux machine so let's close this link you can use it if you want then navigate here to resource group and then click create and choose upon - and here I will create a new resource group called orgy demo rod 1 and I will name the machine VM Linux here the location I will choose you key south I prefer a foster machine with 16 RAM and I choose with username and password the username emerald 1 and I put a strong password and I will leave the port 22 for SS connection opened then click review and create and then click create so this will create the open-toe machine did say navigate back to the resource group just to give some time and click refresh so now I have a resource group and we can see that the VM still in the process just to give a few minutes and it will be ready now it's completed let's say click on the virtual machine and now I will copy the public IP of the machine and open the gate patch then I will try the SSH my name or the username add the public IP click enter the new IPs and then here I will type the password I created while I create the VM now it's complete and let's clear that so if I try a python version we can see version 2 Python installed also Python 3 is installed as well so by default this template has Python 2 and Python 3 installed but if I type in civil and version we can see that NC but is not recognized which means that ansible is not installed in this machine so let's install ansible and configure ansible so let's clear the screen so here first I will update and upgrade the package management for the Linux so to make sure that update all the repository URL and also the package management software so this will update and upgrade the package management for the Linux machine and then I will install here - pipe which is the prerequisite for the anvil library so this will take some times of course I speed up all this and then now I'm going to install the ansible and asher suite so this will install post as well and as your sweet on this virtual machine and this will using the pipe the pre-requisite so now ansible installed so if now let's clear the screen and if I type here ansible version we can see now I have version 2.9 - installed let's clear that so first I need to configure this virtual machine to have authorized the access to adder and in order to do that the first step I need to create a directory dot adder and then create a credentials file and using Nano which is the editor and I will copy the values that I used with the service principles and Azure subscription tenant and ID and secret so I will use it in the credentials file and save that file so ctrl X to exit save double check that my file are saved I will generate SSH public and private keep here so this will generate a private and public key in the default location the private key is ID underscore or SCA and the public key is ID underscore or se the porch is public the passphrase it is recommended to have a strong passphrase but in our demo I will not use any boss phrase I will leave it empty then I need to change the permission for the key then I will use chmod 755 SSH the CH mood changed the mood this restrict the way a file can be accessed and it has different values for the mood for example 755 the file command you allow everyone to read and execute and the owner can the only one to write to this file of course we have different numbers which has different capabilities for example there is number for allowing everyone to read and write there is some to only read and so on so it is based on the number with that permission after I change the permission to the keys are you want to install the skis on the authorized keys and the first step is to change the timestamp of the authorized keys on the server and I will use the touch command in this way touch ssh authorized keys and then changing the permission for authorized keys so I get the permission to I can install a key in the authorized keys by using change mode 644 which means that change authorized keys so we can install the new key 644 only owner can write other can read once I have that now I want to copy the private key to the output and I will use the SSH copy ID and my user name and this will ask me for my password so this will copy the private key to the output so I can copy that you can watch this video for more information about Cooper natus cluster how to deploy docker image to Azure Cooper native service and how to configure CIE CD by appliance for an end-to-end development and deployment scenario in order to make as your pipeline to run the ansible playbook remotely on the ansible VM we need to authenticate a surefire flying to the the ansible VM and in order to do that we will create a service connection on Azure DevOps with the private key to authenticate the agile by applying to the ansible VM so let's navigate to the project settings here PI go for project settings let's open a new instance and we go for the service connection and just create a new service connection and search for SSH and then here click Next and here I will put the private key so let's navigate to the gate bash and copy the private key here with all spaces and copy and paste it here just remove the carriage return and here I will boost the username and the password I use them while I created the virtual machine and here I will just name my connection and civil machine connection so I can distinguish that with different connection here I will put this SH port 22 then I begin to get the virtual machine I be to put it here and then save this service connection so now I have the service connection letsa navigate to the ansible playbook configuration file so I will open the repo and ansible script and open the web app ml which is the ansible configuration file so this is the ansible configuration file which describes the infrastructure as good for our environment or infrastructure on Azure it used the ml format so let's understand how this file is structured so as we can see this is creating the resource group and here creates a service plan and here the backend which is my sequel this is the variables and we see here we have prefix and suffix which will be used so we will replace that using the variables in the pipeline I will show you that later but let's first edit the playbook the first I will change here is a name to be windows because the service plan is Windows not Linux and then I will go here and change the end IP address of the firewall rule to just explain that in order to make my sequel available to be connected by the web app I need to create a firewall rule on my sequel with starting I P all of them 0 and the end of 0 which means that allow other agile resources to connect to that back-end so I need to fix that on the ml file now let's commit this change and now let's navigate now to the pipeline here I will open the continuous integration by applying click edit so as we can see this is were run using the Ubuntu 1604 and the Marvin build will use the poem or the project object model because this is a java application and the poem will explain all the dependencies of the project the restoring the Marvin will restore all the package that listed in the pom file also running all the unit testing the package will be created and many others so the Marvin build will run all that and then here this task will just copy the java application package which is the world file and also the ml file which is the ansible configuration file and infrastructure as code to copy that to be available for the artifacts so this task will search for all the war and the amal and copy them and the next task is to actually copy them to the drop folder on the artifacts in this location the main idea here that we want to store the backage of the web app which is the wall file because this is a java application and also the ansible configuration file or infrastructure as good because the continuous deployment pipeline need two big them in order to understand what is the infrastructure to create and then create the infrastructure then big the web package which is a war file and deploy that package into the web application created so let's now queue a build and I will run the build so this is the builder from outside so let's open a new instance so we can see that from inside and outside let's see that from inside so this is a modern built ask and here from outside so it will create all the any structure in the project object model you can watch this video for more information about git with animation you will see different animated command like tange marriage three days cherry-pick and many others now it's copy the war and the M file or the ansible configuration and here in the artifact let's open that and we can see now I have here this is the ansible configuration file the infrastructure as good the playbook and also I have here the web package which is a war I can also access the artifacts from the build from here under the published so it is the same location so let's navigate now to the release pipeline and select the ANSI well CD and click Edit and click the tasks so as we can see we have three tasks the first one is a replace token so this replace token will search inside my repo for all the ml extension file and start replacing all the token with the variables value in the variable section in the pipeline so let's open that so as we can see it will look at all of these value like location my sake well admin my sequel admin secret and so on and replace that with the you hear and of course to understand it is variable or it is just a keyword by the prefix and suffix in the play book file I need to change this value to put here in the data center that's close to me here I will put my user name this is for my sequel database back-end and here just a password for the database and here just the name of the sequel server this required to be unique so I will just add my name to make sure it's unique and also add my name to the web app then I will get back to the tasks so this is the replacement token as we can see the prefix and the suffix in the words which mean that this is variable this is not a keyword inside the file which will be replaced by the value for these variables the same here for my sequel admin and so on so let's navigate to the tasks next task is the run playbook and civil task this task will run the ansible playbook on the remote and Sybil virtual machine and in order to do that I just need first to identify the connection the service connection to that machine which is the answerable Machine connection that we created earlier using the SSH or private key and as we can see this will be a remote machine but the source of the playbook will be the current machine the agent machine because the pipeline will pick up the ansible playbook configuration file or the infrastructure as code from the artifacts and download that on the pipeline and here this is I just need to choose here host lists this is the way that I'm going to run my playbook and here in the hosts list I will put the IP of the remote and Sybil virtual machine so let's copy the public IP of that machine get back here and just put it now this is complete let's go for the third task as your app service deploy this is the tasks that will deploy the wall or the Java with application package into the whip app created by the ansible playbook so first I need to authorized my pipeline to the azure so it can deploy that so this will just authorize and login with my username and password just type my password and just give few second so this now authorized and then here I will just as we can see here set the app settings in the web app configuration so I using here the value in the variables in the pipeline like the the database username and the password and the web app name so I can configure the connection string in the app settings all of these values from the variables which will be created the resources so now it's ready and remember it is using hosted visual suite 2017 because the app service deployment will use the MS deploy which is run on a Windows machine not on a Linux machine the PlayBook will run on the remote Linux machine but the MS to blow it will run on the agent machine which require Windows machine so now let's see if that and before create a new release let's navigate to the azure resource to look at the current as a resources so let's get back here and click resource groups so this is the current resource groups let's refresh now get the pipeline and create a new release and here I will select the previous build that I used in the previous task which published the web package war file and the ansible playbook on the artifacts so the pipeline can understand from where to pick up the package and the ansible playbook configuration file let's create and now let's open a new release view in essence two releases one from outside and want to see that from inside and I click here so this is outside we can see one of three tasks and here this is the inside so here to just download the ansible and the way package and now start running the PlayBook remotely on the machine so as we can see the pipeline starts downloading the artifacts which is the playbook on the agent machine and running the NFL PlayBook remotely on the Linux virtual machine and the first step here is to create the resource group this is from outside it's four or five tasks so as we can see here create the service plan so let's navigate to the resource group so we can see the created resource group let's click refresh as we can see now the NCL resource group created if I navigate here we can see the service plan even created which is a Windows service plan the infrastructure for the web app it's a navigate back here to the pipeline now it's creating my sequel server this will take some times let's refresh now as we can see now it's created the azure web app but my sequel is not created yet it is still in the process of creating the back end of my sequel just to give some time of course I speed all of that now it's created and working on the firewall rule and also creating the database on my sequel server database and now it's deploying the application so all the infrastructure created and now the deployment the war file into the web application so this deploying all the war file extracting them now it's completed so if I navigate back here refresh so now we can see that I have my sequel database created this is my sequel and also as we saw the firewall rule opened so if I open here the web app and navigate to the configuration and we can see that I have now all the configuration settings this is the password I put it let's navigate outside and if I open now my sequel to see the firewall rule connection security we can see the firewall rule is on because usually by default if I didn't creating this firewall rule it will be off which means that the web app will not be able to access my sequel database it's an advocate back here and open my web app and click browse so it will take some times and then loading the web app with the back end on my sequel so let's here put the username me and smart hotel with password 1 2 3 4 click login so now I login to the application which is a smart hotel and we can see the application is running loading the data from the database and it looks fine at the end I would like to thank you for watching the video please if you have any question or comment don't hesitate to reach out you can find more information on my blog which will appear at the end of the video along with some other related video thank you [Music]

Channel: Mohamed Radwan - DevOps

Views: 13,154

Keywords: Ansible, Infrastructure as Code, Azure Pipelines, Ansible remote machine, provisioning infrastructure, Azure, Continuous Integration, Continuous Deployment, CICD, CI/CD, CICD Process, DevOps, Build Automation, Install Ansible

Id: Q8aWeHCrGh4

Length: 34min 40sec (2080 seconds)

Published: Sun Dec 29 2019