LetsDevOps: Parallel Processing - A Guide to PowerShell Jobs. How to run ARM templates in Parallel

hello everyone welcome to Let's devops I  am Sumit and today we are going to discuss   about parallel processing using Power Cell  further we will learn how to run a multiple   arm templates or a command in parallel using  Power Cell job for all the demo and discussion   I'll be following this article you can find  the link of this article in video description   so let's start with introduction so in this  article first we will learn the basic of   Powershell job how we can run it and further we  will set up a multiple airm template execution in   a parallel using Power Cell job let's understand  the use case so we had a requirement to build   multiple Azure resources like a VMware storage  account using arm template so in the beginning   we tried using a power cell to call those arm  templates sequentially and that was taking a lot   of time because it has to wait for the second  execution because it was running sequentially   so when we used to create a multiple VM like  a 10 or 20 it used to take a lot of time right   because if you run sequentially once the first VM  gets created then it will trigger the second one   and that part we need to improve keeping this use  case in a mind so we need to do an improvement   right so if you have options to run execution  in parallel we can achieve that right so let's   explore the option of Powershell job Powershell  job helps to run multiple command simultaneously   in a separate process or a thread so command can  be anything command can be maybe you are calling   another script or maybe you are running set of  script itself and everything that you want to run   in a background you don't want to interrupt the  console of our parcel or execution you just wanted   to proceed and run all the process you wanted  to execute as a background then you basically   create a parsing job now coming to why and when  to use a power cell job so this is helpful for   automating a repetitive task and improving the  performance so it helps to run a script or a task   in a background without blocking a parcel console  or a terminal so basically if you want to run if   you want to run your code snippet 10 times you  just need to invoke 10 times in a background and   it will all run in a parallel ok so let us talk  about the job architecture so if you see here   in in a in a computer there will be a power cell  terminal right and whenever you create a job   it can basically create a separate session so  assume that I want to create three different job I   will basically create three different session and  all those systems can run independently and also   the parallel so yeah so assume that I have one  the main power cell script and under that I want   to call specific command or another deployment  then I'll create a job and that job can be may be   created multiple times like multiple job you can  create so basically assume that in this example   we have created three different job and  these three zero pin shot will run parallely   okay now let's understand what are the different  command we have in a Powershell job the first is   start job with the help of this particular command  itself we create a background job in a new session   on a local computer get job basically it will  give the status of your job which you created   receive jobs it gives the result of your  background run like once it gets completed   your execution gets completed and you want to see  the logs or the detail you just use the received   job stop job itself again it says that if you  want to stop your background job you just use it   where job is to you you will basically if you  want to suppress the command prompt until the   job completed then we use feed job and remove  job again is just to remove your background job   so since we talked about the start job which  basically runs on a local computer in the sense   maybe if you are running on an agent it will run  on the agent but assume that if I want to invoke   some of the backgrounds on in a remote system  like basically after I'm running from my local   and I want to invoke a background job on on a  VM or on remote VM on a cloud then I need to   use the invoke command with the help of start  the job so with the help of this combination   you will be able to create a background job  remotely okay so now let's understand uh the   example and then I will also show you how it  runs in a local so if I say that okay start   job and then if I say script block whatever the  script you defined it will run in a background   and again the get job is to get job and then  read job with the ID you will see this the   execution results okay so we will now try  all the command one by one and here I have   given this code snippet you can uh maybe the the  logs so you can also try on your side right now   let's try to create some a job the the Powershell  background job and let's see how it works okay so   I have opened uh the Powershell command and then  I will try to just use a job and then start job   and then I will use a script block where  I can Define the command that I wanted to   run as part of the background process so let me  try to run the get process so basically when we   run it it will try to run this particular  command get processed in the background   so right now if you see that the console  came out right so although it is running   it doesn't wait for that it didn't wait  for that and it just comes out it means   that that particular job is running  in the background so let's try to see   status of the job so if I say get job now if  you see it's it is running as a background   job here you can find the detail of your job  like job ID is five that is a job name and then   this is a background job type and currently  it is running okay so let's try to get job so assume that if I want to see the status the job okay so now right now you see it is just displayed  right so whatever it came out now let's again try   to see the get job and let's see the status so now  if you see the status now it is completed it means   that the job which was running in the background  it's already completed and if I want to basically   remove this particular job just I will need  to do that to get job and then remove go sorry so now again I'll look at job so I should not  not see any job now yeah it's coming as a blank   so this is some of the basic command that we  can use it while creating a background job okay so since we use the script block while creating  our job so let's understand what is the script log   so script lock helps to define a command you  want to execute as part of the background job   you can basically run or Define multiple  commands as well so if you notice in this   particular the job I have defined a Blog and I  have added three different commands the first   is the write host second is the get Services  where I'm trying to fetch the service which is   running which is in the running State and also  I'm trying to find the services which is in the   stop state right so let's try to run this  okay in our Command so I just copy this for sale okay so it ran already you  know let's try to connect so it's also completed so  let's see the the full log okay so now if you see it is displaying  all the list of services which we have   which is in the running State as well as in  stock State and also it should say the the   print right running multiple command under  a single job so this is the the right host   which we added in a command like this one right  so basically this is how we use the script block   to Define we can Define multiple commands  in it and then you can basically execute uh   in a background okay now let's talk about the  argument list so augment list is very important   for us because as we know that when we create  a job basically it creates a new power cell   session and when it creates a new power session  it doesn't contain or hold any variable that   you created from a parent right so basically when  we create a job it create a new session and that   does not hold any a variable or any context or a  setting that you set it up in the parent session   so assume that in this scenario if you want to  pass some of the value as a parameter then in this   case you use argument list so if you see in this  example I have declared two different variable   value one and value two and then I want to  basically contain or use this particular value you   know script program in a background job in this  case I need to pass that value as an argument list   and with the helper argument 0 and the array of  arguments one and two we can pass like this okay   so first we maybe use it as a value one value two  we Define a variable and then when we create a job   we pass those value as a argument list and this is  how we use it's very important because sometime if   you need to pass some information from your parent  session to the child one then in that case you   need argument list itself to pass that information  okay now let us try to run this in a power cell   and then see how it works okay so I open uh  the power cell ISC and let me paste it here   okay so before uh running like argument list  let's try to not to pass as argument list let's   try to remove this and see whether it is printing  it or not okay so assume that in the I want to   basically display this to variable value which is  uh I am calling from this particular system itself so now if I run this basically this two value is  getting stored in this particular session right   and when I run this the job the start job  as a background since it will create a new   system it will not hold the information of this  so let's try to run and see okay so I'll run this   right now if you see here it already ran so let  me try to do a get job and then receive your   job here we go see the value is coming as  empty right because we have defined this   value in this particular session and whenever we  are creating a job it is creating a new session   which will not have the value so in this case  only we use argument list let me set it up now   and then we we basically pass as a parameter  the way we use to do it with a function   and then here instead of value  one you see the taller arcs 0 and this is a okay now let's run this okay now see now it it contains  the information because we are   passing this as a argument list okay so now we have understood the basic of the power  cell job what are the different command we have   how we use a script block how we use argument  list let's see how we can basically you make   use of this particular uh the feature from  Power Cell to implement or to run multiple   arm template basically that is our use case and  then we run all of the arm templates parallely   so yeah for a solution we are going to use the  Powershell job and this is the architecture so if   you notice here we are creating multiple parameter  file during runtime and we have one template   file that is basically getting called with each  parameter file and that combination of each param   with a template file we are creating a job and  all the job can run parallely assume that I need   to create a three different VM right I'll create  three different parameter file and then whenever   I am calling with the template I am running  basically as part of the job okay so for three   different VM maybe three different job we have  and all of them can run parallely independently   in this case the time of creating the VM will  take less right compared to running sequentially   okay so this is the architecture which we  implemented maybe in your case you will have   a separate parameter file or template 5 also that  is also possible you can assume that if I want to   run the template for maybe storage account VM all  together that also can be doable but this is the   use case which we had where we need to operate  the parameter file for each VM and then we need   to call through the arm templates template file  and we used to create multiple job in parallel   and that's how we create maybe 10 or 20 VMS all  running in a parallel within a 5 or less than   five minutes okay so if you want to try this you  just need to go to the Aram template here and here   you can see there are three different files one is  this power cell file second is the parameter.json   and third is the template transition so in the  parameter.json if you notice I have tokenized   the storage account name in your case maybe if  you want to tokenize some different value that   also you can tokenize and as part of the power  cell script first we update this parameter file   with the tokenized value and then we run it based  on our requirement so let's try to open this power   cell here we are just giving the resource Group  name under which you wanted to create the resource   and then template file you can always update  if you want to store in your local and you're   running from your local that you can give if you  want to store in your repo that also you can give   a path so as of now I have just hard coded  that path and the parameter file base is like   nothing but the the file or the folder where your  parameter file will get created dynamically or   during the runtime because you have tokenize the  parameter file and if based on your uh the count   you'll basically needs to create those parameter  file first and then it will run the arm template   so this is the the for like customer if I  want to run this particular arom to click   five times then I'll just give an account as a  five and here if you notice first I am trying to   create a different deployment name also because  that is required and then I'm updating the the   tokenized value and creating a the parameter  file new parameter file after updating the   tokenized value and then we are running the the  background process so since here I've ran it on   my local so I didn't need to provide the connect  result and all but that also you can specify in   your case and that example I have given in the  appendix so here you can always configure that   okay so that's all for the today's discussion if  you have any issue in your configuration you can   write in your comment box I will be happy  to help thank you thanks for watching bye

Channel: Sumit Raj

Views: 683

Keywords: LetsDevOps

Id: 0eV_xBdirEo

Length: 17min 0sec (1020 seconds)

Published: Sun Apr 16 2023