PowerShell Tutorials : ForEach-Object -Parallel (Parallelism in PowerShell)

hi and welcome to this powershell tutorial video in this video we're going to be looking at running scripts in parallel so microsoft has made it quite a lot easier to run tasks concurrently or in parallel um you can call it as well with powershell 7. now you've been able to do this since powershell version 2 i believe but it was always involving quite a lot of work either creating workflows or creating run spaces and then in powershell 6 you were able to do start dashbread jobs but then always had to remember to also remove them so with powershell 7 they've added a really cool parameter on the for each for each object it will not work if you only type for each but if you do type in for each object you are able to add the parameter of parallel and it will only take in inputs from a pipeline so we're going to be taking a look at that today to see when you should be using it and what type of benefits does it bring you and where cases that you should not be using it because it's just strictly going to take more time because of the increased overhead of creating those run spaces in the background so let's go ahead and let's just start to see how we can actually run in parallel here so i'm going to create a array here i'm just going to do a dollar sign array and i'm going to make my array just equal 1 to 10. so this is like a short form of creating an array from numbers 1 to 10. so if i just run this here and then we look at what is inside of array we have numbers 1 through 10 so then what we're going to want to go ahead and do is we're going to want to go ahead and loop through this array so typically what we could do is we could do array and then we pipe this into a for each right for each object we're going to do for each object right away then we can do an open and close curly bracket and then we could do a write output dollar sign underscore to take in the input from the pipeline so let's run this here and we are going to see that it simply just prints out 1 through 10 and then to see that a little bit more visually what we can do is just add the word i'm in front of it so there it is i'm one two three four five six seven eight nine ten now what we could do is we can add at this point our parameter which is just called parallel and then if we run this again it would help if i spelt it correctly so there we are um so now we can see that so i'm just going to expand this out a little bit just because this is one of the first main points so without the without the parallel parameter uh we can see that we get one two three four five six seven eight nine ten now when we add the parallel parameter we get two one three five six seven four eight nine ten and this is because each each time we're just creating a thread and we don't care about in which order they they print out we are just telling parish uh powershell that we want to run all these options from the array in parallel as fast as you can and then the computer then organizes all the threads per the amount of cores that you have on your computer now this doesn't mean if you just have like a an eight core processor it would only run eight times there are logical processors so usually in a two-core system you would have about four logical processors in a four-core system you would usually have about eight now this could vary per system and also if you're running it on a server or not um so it would really depend on the type of processor you have and whether or not it is a server or just a workstation uh but also you can run multiple threads on one core it'll just block itself and then take turns and then in fact seemingly look like it's running concurrently i'm not sure if that's 100 clear or not but this is how you would run things in parallel now let's see the performance difference from the two so if we do a measure command here in front of these arrays um so in front of these right outputs so we're gonna do um one with parallel so let's just see what this takes so currently to run this brew in a ray of 10 it is taking milliseconds with the parallel parameter without the parallel parameter it is taking 18 milliseconds so we can already see that in the case where we're just printing parallel isn't faster and we can actually make this a lot worse is if we have a thousand entries in our array if we're just running them serially so one after another it is taking 58 milliseconds if we add the parallel parameter it is going to take us two whole seconds and 422 milliseconds so much much longer and this is because when powershell gets before each object parallel it will start up a a run space and then run the inside of that for each object loop in another thread and it does that multiple times over increasing memory usage increasing um like the processing power that your computer is currently using and might not necessarily be faster because it takes time to spin up those run spaces compared to just printing an i'm and then a number now in cases that this will be useful because of course i'm not going to show you guys just where it's not useful but definitely one of the cases where it's not useful is if you're just simply printing to a screen or doing very very basic tasks or if you're running a vm with only one core the parallel keyword will not do any difference because you only have one core you'll probably actually just make the performance a lot worse because you're spinning up all these different run spaces and causing a lot of blocking in those spreads so it taking a lot more time because at that point then to complete one of the threads it'll be limited because it's got less resources now to complete that one instead of going serially where it could dedicate all of its power uh to one at a time making it faster now one of the times where it will be much faster is if you have something uh that needs a waiting period now we can simulate this so this would be something like an api call or another commandlet like getting the event logs which we'll actually take a look at in this video as well uh to see what the performance differences are so we can actually simulate this by doing uh we're gonna just put in as the first oh what we're gonna do entering and then the number and then we're gonna do a start leap and we're gonna start sleep for one second and then we're gonna do the right output and we're just going to do exiting and then our number now what i'm going to do is i'm going to take away the measure command for now because i just want to show you guys how this actually executes and you guys will see how the threads are actually entering and exiting so here we have just an array of 15 just to make it go a little bit faster so let's see what this does all right so here we have we have entering one two three four and five and it exits out of one and two so it creates two more threads at six and seven it finishes the bird thread so it then starts number eight and it exits so this one we can see that it's only using five breads uh which is perfectly uh perfectly fine but you can actually modify this using the throttle limit so if i do the throttle limit of three we are going to see that it is only starting three jobs at any time so it will not run more than three threads at a time and we can change this to four and then we will see also the other difference so here we are we have it at four so the first it starts up the four threads it exits out of number one so it starts up another one for five exits out of number two so it starts up the next one for number six so as you can see uh you can really control this so if you do not want it to take so many cores to not jam up your computer resources very very quickly you could do it this way and we can even do a throttle limit of 10. so there we are it shows up through 10 and then it starts finishing and then it does some more as well so let's see how this actually turns out if i take away the throttle limit for now and i do a measure command so now we have our measure command with our parallel command here so let's see how long this takes to run so we're we have 15 elements in our array and we are sleeping for one second for each but we see that the actual whole command only takes just under four seconds to uh just over three seconds to run so three seconds and 93 milliseconds so pretty much three seconds there now if we actually remove this parallel keyword and we run this we will see that it takes much much longer because it's gotta execute each one and it's gotta wait one second each time so in here we can actually see where the benefits are of using this for each object parallel commandlet it's going to be in scenarios where there's a lot of waiting involved now we simulated the waiting with a start sleep commandlet um but we're going to simulate it now with a get win event now this is not going to be as extreme because we're only playing with one computer and we're only going to be doing maybe three or four event logs but this could definitely get quite expansive if your process takes let's say five minutes uh to execute you're doing some api calls you're going to be doing some stuff with that data and all in all that script for each one normally takes about five minutes and you have five of those so you have five times five minutes if you're running those it's 25 minutes if you have 10 of those it jumps up to 50 minutes but if you had a for each object parallel and you executed that task with a throttle limit of let's say five it's going to be running five of those at a time it's only going to take about five minutes uh give or take maybe a little bit more maybe maybe a little bit more because of the creation of the run spaces but that's what it would really end up taking it would launch the five threads that each take about five minutes but they're all running at the same time meaning that it'll only take five minutes and if you have 10 it'll only take 10 minutes compared to the 50 minutes so those can definitely get some exponential savings when you're dealing with numbers like that these numbers are quite small because we're just talking about a one second waiting period but let's go ahead and let's take a look at a collection of logs so with the event viewer so what we're going to want to do is let's go ahead and let's just take away all these variables here and we're going to be looking in just a few of the logs so we're going to create a array called log names and we're going to create this as a security application then in for the next one we're going to do windows powershell and then we're going to look up the dns server logs as well so here we have all of our log names so let's go ahead and let's do log names and we're going to pipe this to for each object and let's just do uh the regular for each object at first so then what we're going to do is we're going to do events equals get win event for the log name and then we're going to pass in our log name here again it's just the dollar sign underscore because we are taking it from the piping of log names and we're going to do max events let's get um 000 actually let's get 10 000 events at most and then what we're going to do is so that's all we're going to do for right now we're just going to simply collect those events and see how long it takes without the measure command just to see for fun how long it takes so here we have it it's taking a little bit of time we've seen this get win event before in a previous video and i've shown you guys how to make it a little bit faster with the filters with xml but i am purposely just making it take a little bit longer here all right so it took a decent amount of time so what we're going to go ahead and do is we are going to do a measure command and we're going to open that there and we're going to close that all right so let's see how long it takes again this is not running in parallel so let's see the results here and this should take so this takes about 10 seconds uh so it's not bad it's not super fast um but it's not super long now if we add the parallel keyword to this and we run this now we can see that this took only eight seconds and 344 milliseconds so a two second difference on just collecting a very very small amount of data this computer does not have a lot of events in the logs so it's not a great amount but you can definitely see and if we were collecting these from remote computers or doing other things with these events it would definitely be a lot more time saving and i mean two seconds if we extrapolate that throughout like an entire day um it could be quite long depending on how often you are running these commands now what if we wanted to store these events into a array now this is where the for each parallel does make it a little bit more difficult uh because you cannot access outside variables very similar to powershell remoting when it creates a run space it is a isolated bread so it cannot access the state or variables of other reds so what we need to do in order to be able to store these events so what we would typically do is we would have a export variable here and we would typically create it as a system dot collections dot array list and create that as a array list here and then what we would typically do is do export dot add and then events and then we would print out the export so if we actually try this here we actually get an error because we cannot call a method on a null valued expression now and this is on the line three export dot add events now it's determining that export is actually null which we know it isn't because we have the arraylist here but because of the for each parallel in the thread that this is running export does not actually exist yet so in order to import this variable very similar to how we did the powershell remoting we need to use a keyword so what i like to do is i like to recreate the variable in here called export and we're going to create that and then we're going to make that equal to dollar sign using and we're going to do a colon and then export so what this is telling it it is using export from the original script here the external value the variable externally here so what we could do is we could then go ahead and run this and see what happens we can see that we don't get the errors and then when we go to print out our export at the end we will actually see that it does print out all the events so everything works properly so that is a little bit of a gotcha there in the for each object if you are planning on doing some log collecting in order to uh bring it back into your script later on and all you really want to do is collect them all in parallel so your script goes a lot faster but then you want to manipulate them later in another area this is the way that you would want to do it now there is one thing is that this array list here is actually not bread safe so you might get errors saying that it cannot write to that arraylist because it is currently blocked by another process so in order to actually facilitate using an arraylist you have to make sure that it is synchronized and i'm gonna put a link in the description down below kind of explaining what synchronized is uh it just makes sure that it is a thread safe version of the arraylist so it doesn't get blocked by any threads so what we're gonna wanna do is we're going to want to do the same thing our export is going to be equal to system.collections.arraylist then we're going to do a colon colon and then we're going to do synchronized open parenthesis open parentheses and it's going to be a new object system dot collections dot array list and then close parentheses and close parentheses so now if we run this we're going to see that it looks exactly the same but in the background this is a breadsafe arraylist making it sure that we are not going to have we are not going to encounter any issues within our for loop uh for this arraylist so that is really really important to note is that any variable like a dictionary or a hash table or arraylist and you are modifying it within your for each object uh parallel or through bread jobs you are always going to want to make sure that you are using a synchronized table or synchronized list or synchronized hash table to make sure that everything stays intact and you are not causing yourself potentially problems down the line of missing values in those variables because of a lock that had happened and again i'm going to be putting something in the description down below which is an article from microsoft that just describes the synchronized arraylist for you guys as well so now there is one other gotcha in that is if you were wanting to use a function now to use functions within a for each object it is a little bit trickier um so what i'm going to do is i'm going to go ahead and i'm going to create a function up here and we're just going to go ahead and call it say hello and we're going to do an open and close bracket here and all we are going to do is do a write output and we are just going to say hello all right and then what we're going to want to do is we are going to have a array here and we're going to make it equal to just an array of five and we're going to do array we're going to pipe that to for each object and we're going to pass it in parallel and what we're going to do is we're going to do a say hello so in theory what we should see here is we should see the output of hello five times so if we run this here we actually get an error saying that the say hello is not a recognized as the name of a commandlet or function and that is because once again it cannot access outside of this this function is not declared inside of the bread of the for each object so what we actually need to do is we need to create that connection for that for that function so what we're going to want to go ahead and do is first create a function definition outside here so we're going to just create it as a funk death variable here and we're going to make that equal to function colon and we're going to call this to we're going to have to reference our function which is say hello and we're gonna cast that to two string and cl open and close parentheses then inside of the for each loop what we're gonna come and do is we're gonna do another function colon and we are going to call it the say hello function and we are going to make it equal to using and we have to use our function definition here and then what we can do is we can then call the say hello function so if we run this we actually do see the hello five times so we do see that it is working so there are definitely some uh tricky parts to the for each object parallel especially when you want to incorporate some external functions or some external variables within that for each object if it is a fully self-contained for each object like it's just going through the data making some api calls and then performing some action on the system it's probably a best case scenario you're not going to really have to do any extra work to bring in something but if you do have some logging involved where you are logging to a log variable that will get exported at the end of your script in that case you will need to do a little bit of that work of using the using keyword so i would always just make sure uh that if you are referencing something externally make sure you just have those connections made or try to always if you're using the for each object parallel try to keep it all self-contained and this way you don't really have to worry about that but of course there are some scenarios where that cannot happen but that'll be okay um so that would be really the main part of for each object with the parallel keyword now i actually have some examples where we might think that the parallel keyword is going to be very useful but it actually turns out to not be useful so i'm actually just going to paste these examples in here so what we've had in the last video was a csv um some csv files that we were going through adding the sum and then pumping them out to the display and actually this you would think that it would benefit from the parallel keyword but it actually does not so here's our script uh from last time let's do it without the parallel keyword at first uh so we're just grabbing the csv files and then we're grabbing the data and we are outputting the total so let's just see what it looks like without the parallel keyword and if we run it without the parallel keyword with the measure command we can actually see it only takes 60 milliseconds which is really really fast and then we can add the parallel keyword here and if we measure the command here now we see that it's 50 milliseconds so it is actually quite significantly slower almost twice as slow almost three times as slow compared to running it serially so what i always recommend is if you have a script that is running slow in serial maybe try to run it in parallel c if you get that performance increase that you want um but in a lot of cases running as parallel will probably not make things faster if the script is already pretty fast um if you are running things on a single um core or single cpu vm i definitely would advise against running in parallel as it will not have any benefit to you at all um if it's very simple tasks like printing or reading data i would say it's probably going to be slower to do a parallel for each object because of the amount of time and the amount of resources it takes to create those run spaces but if you encounter situations where it's a very time intensive task like getting event logs uh maybe filtering them a little bit further we've seen some examples in the other videos of getting event logs taking quite quite a bit of time sometimes a get event log could take two to three minutes so if you did have five of those logs that you were getting that are taking two to three minutes each in that case you would have a big big benefit of using the parallel keyword with the for each object so it's really just to know when and where to use the parallel keyword or the parallel parameter on the for each object it is very tempting to do it for everything but there's actually a lot of situations where the parallel parameter is actually not beneficial another example that i can give uh would be a active directory user creation uh it might actually seem like it is a lot faster if you were to run it in parallel but i actually have an example here so let me just copy this example in here that we have so we have a csv file that i have here with five different accounts and that's just five accounts um so we import the csv file and we create a ad user or each of those lines so let's actually see how long let me just make sure that these users actually are not still in my active directory which they actually are so let me just delete these users here before we do our test and let's do the measure command so this is running not in parallel so this is running in serial creating one at a time so let's see what happens uh oh and actually that will not work because i did not put in the correct folder name uh so let's run this again here let's just make sure the users got created which they did so we can see that it only took 381 milliseconds to create these users uh so not very long at all so if we actually delete these users once again and then we run the task as parallel here so we can already see that it's actually already taking a longer time and now when we look at the time it actually took five seconds and 697 milliseconds so that's almost an increase by about six times uh the amount of time actually i think it's a lot more than six times it's it's a lot more um because if i look at the milliseconds it is 5697 compared to 381 um so that is about like i would say let me actually just get out a calculator here um so that is 500 and uh 5 900 697 divided by 381 so it is actually almost 15 times slower running it in parallel because the actual command of new ad user does not take a lot of time or a lot of processing power the computer is not waiting on active directory to then go on to the next one um it will actually create that new ad user send that command and it's already on to the next one whereas right now it's doing a it's creating the run space it is creating this hash table sending the new ad user and sending that off and then doing that for each one so it has to create five run spaces uh where it just doesn't really need to in this in this situation another situation where actually ad user i would actually advise against doing parallel is if we remember our script where we've automated users we were checking at the time to see if the username already was existing now if you're having five threads going at once you might encounter the situation where that script tries to create a user with the same username because the first time when it checked it didn't have a user there but then one of the threads so you'll have your two threads creating a smith uh one uh person's name is john smith and the other person's name is jane smith and they both look at active directory to go see if the username smith j is existing but both of them haven't executed the new ad user command so they'll both get that the smith j is available to use and once it actually pipes that into the new ad user um you will actually get two 80 users that are going to try to get created as smith j which one of them is going to error out um and potentially you might miss that or it'll cause an error in the script and then you'll have to go back and fix it uh creating more of a time loss and we've already seen that before each parallel already runs a little bit slower so in all uh this would just be a very very bad way to use the for each object parallel uh so the two cases where again that i would use it the most would be api calls and event viewers of course there are a lot of other options that you can do with the for each object parallel that i did not mention here the always the best way to see if it would benefit your situation is to test it out with the measure command and see if there is any benefit and if there is a benefit then use it but just be aware that you should not be using it for every scenario as it is not beneficial in every scenario so i hope that helped you guys with the for each object parallel command line i might have repeated some information over and over again um but i just want to be sure because this is quite a topic that even i struggled with uh quite a bit to learn in the past um knowing that why is it not beneficial in every situation i figured that it would always be faster to run things in parallel but then learning that that is not always the case and with this measure command it really kind of shows you how that works so if you guys have any questions or comments leave them down below in the comment section and i will get back to you if you guys have any questions um and want to see a video on that topic let me know down below in the comments section as well hit like and subscribe and hit that notification bell to be notified when that next video comes out and i will see you guys on the next video

Channel: JackedProgrammer

Views: 6,401

Keywords: powershell basics, powershell for beginners, learn powershell, powershell script, powershell scripting, powershell, programming, scripting, coding, automation, powershell tutorial, parallel programming, powershell parallel, foreach-object parallel, foreach-object parallel variables, foreach-object parallel functions

Id: w_4Slu19DcY

Length: 36min 22sec (2182 seconds)

Published: Thu Feb 24 2022