Synchronizing Multiple Processes in Python

what is going on guys welcome back in this video today we're going to learn how to synchronize multiple processes when doing parallel Computing in Python so let us get right into it [Music] alright so we're going to learn how to synchronize and coordinate multiple processes in Python today now we all know that python does not support actual multi-threading because of the global interpreter lock however python does support actual multi-processing which means that we can use multiple different processes to work at the same problem at the same time in parallel and whenever we have something like this we usually need to introduce some coordination and synchronization because we have shared resources shared values that are accessed by all these processes at the same time in parallel and the problem is when one process tries to write something another process tries to read something information might be lost and then the result is incorrect and inconsistent and not predictable so in this video today I want to show you a simple example of a counter being increased and decreased by multiple different processes and I want to show you what happens when we do use synchronization and when we don't use synchronization so I'm going to start here by importing multi-processing and I'm also going to import time to slow things down a little bit and then we're going to Define two functions the first one being increase counter and it's going to take the counter as an argument come on stop mistyping here um increase counter and what we're going to do here is we're just going to say four placeholder in range and what we want to do here 20 times is we want to say counter dot value plus equals 10 for example uh and then we want to sleep so time dot sleep for 0.1 seconds actually inside of the loop so that's basically what we want to do here 20 times we want to increase the counter by 10 and this is the purpose of this function here now then we're also going to have maybe I should just copy this here I want to have a function decrease counter and this function is going to do roughly the same thing it's just going to subtract but not 10 let's go with 50 maybe and let's also make it a little bit slower just so we have some difference here so 0.3 seconds maybe um and those are the two functions that we want to run in parallel now with multiple processes so what we're going to do first is we're going to define the counter that we're going to pass these functions so the counter is going to be equal to and now since multi-processing is not the same thing as multi-threading we don't have the same shared memory space we need to use an actual value here a multi-processing value by defining it as a multiprocessing DOT value and what we pass here as a parameter is first of all the data type I for integer for number and then 0 as the default starting value and what we're going to do now is we're going to say here increase underscore processes is going to be an empty list and decrease processes is going to also be an empty list the reason we want to have a list or we want to have two lists for those processes is because we want to also wait for them before we actually print the final result because whenever we we start new processes the same main process here that starts these processes continues to run and we only want to print the result of the counter after all the processes have finished so what we're going to do now is we're going to say four placeholder in range 5. we want to create a new process P so five processes for increasing five processes for decreasing I want to say p equals multi-processing dot process actually with a capital P the target of this process is going to be the increase processes function and the arguments for now it's just going to be uh the counter with a comma to indicate that this is a tuple then we're going to say p start to actually start the process and then we're going to add this process to the increase processes list so we're going to append it then I can copy this here and what's the problem here uh oh sorry I wanted to actually have the increased counter function not to increase processes list so here to decrease counter function and we don't call the function we pass it as an entity and here we basically do the same thing with the decrease processes come on decrease processes append P so five for increasing five for decreasing and what we want to do now this is why we have the list is we want to say 4p in increase processes P dot join so wait for them to finish do the same thing for the decrease processes and only after all of these processes are done print the value of the counter so you know you can calculate what should uh what you should get here as the output but I can run this now and after a while when this is finished it gives me negative three thousand four hundred I can run this again now and see what happens and I'm going to get let's see negative 3850 and I can run this a couple of times and I'm going to get different values even though obviously what I'm doing here is every time the same thing I mean I have five processes that increase the number 20 times by 10 and I have five processes to decrease the number 20 times by 50. sorry so why do I get different results and the reason is because there is no synchronization things are happening at the same time processes are writing processes are reading and you know they are not synchronized so while a process is writing another one is reading information is being lost and we get inconsistent results so what we can do here now is we can add a parameter here log to the increase counter function and log to the decrease counter function and uh what we want to do now is we want to create here also a lock which is going to be a multi-processing Dot Lock and this lock now we can acquire this log and we can release the log and it can only be acquired once so if you acquire the log you have to release it before someone else can acquire the log again so what we're going to do now is we're going to say every time we want to change the value we're going to first say log dot log dot acquire like this and when we're done we're going to say actually um actually here lock dot release and we're going to do the same thing down here and this is going to synchronize everything now the only thing that we need to add here now is the argument lock to the process and then we can run this and you will see that we will always get the same result which is negative four thousand this is what you should get here as an output and you will always get this if you use synchronizations if you use synchronization or if you don't have multiple processes now another way to write this is to not say acquire and release you can use a context manager here you can just say with lock and with lock basically does the acquiring and the releasing for you um for the indented code so it acquires the lock it executes the code that is indented here and it then releases the lock afterwards so we can also say just with lock here and I can run this again and you will see we get negative 4 000 as a result again there you go so this is why you need to synchronize access to Shared resources and when you have shared resources across multiple processes you use the dot value there's also dot array and stuff like this but you use dot value here in this case and you use a lock to synchronize the processes now I also have videos on this channel on threading and on uh actually synchronizing applications because with threading you have very similar structure you don't need to have these multi-processing values because the values in a uh or in in threats are in a shared memory space so we can just access them normally but you need to have logs as well and when you have multiple applications that maybe have a different code and don't have are not related in any way but they access the same file you can use file locks and I have a video on this channel where I explain how to do that as well so that's it for today's video I hope you enjoyed it and hope you learned something if so let me know by hitting the like button and leaving a comment in the comment section down below and of course don't forget to subscribe to this Channel and hit the notification Bell to not miss a single future video for free other than that thank you much for watching see you in the next video and bye

Channel: NeuralNine

Views: 9,718

Keywords: multiprocessing, synchronization, coordination, multithreading, python, locking, locks, shared values, python multiprocessing, multiprocess synchronization python, python synchronize multiple processes, multiprocessing locks python

Id: 8wX1uEp2Jhs

Length: 9min 31sec (571 seconds)

Published: Wed Sep 13 2023