Functional Programming in Python: Parallel Processing with "multiprocessing"

hey and welcome to the next video in my functional programming in Python series so in this video we're gonna talk about parallel programming how can you execute code and do data processing in parallel using Python and using functional programming principles so in this video we're gonna take everything you've learned so far and use these techniques to very easily it take some existing code and make it parallel so that it runs on all of your CPU cores at the same time potentially you're reducing the amount of time it takes for your program to run and I'm trying to do this with some very hands-on examples that build exactly on what you've seen in the previous videos so what I've got here is a simple test bed that brings back our data structure here so we've got this immutable data structure that represents a bunch of scientists that worked in different fields when they were born their names whether or not they won the Nobel Prize at some point and then we've got a simple transformation here where we take each scientist and then create a transformed output data structure that contains the scientists name and their age as of 2017 and all of this is going to look familiar so at this point you probably want to check out the first video in this series on how and why we set up this data structure the way it looks like here and of course also here that's about the named tuple and then you probably won't also want to watch the video on the map function that that showed you how we can take this scientist structure here and transform it into a new data structure all right so let's run this example because I think that's gonna make it a little bit easier to follow I save this as parallel PI and when I run this you can see here well okay we're just printing out the input data structure it's a scientist thing here and it looks like this and then we're applying the transformation using the map function and we're printing out the result and this is the result and that we get here so we get all of the ages for these scientists and their names now this transform function here is really simple right it's we're working with a relatively small amount of data here and we're working with a simple transformation that doesn't take a lot of computations and it will execute very fast but if you imagine this was a more complex operation here for example if this needed to go out and fetch some data from the internet and then process it so if it was IO bound it was waiting for IO to complete for that website fetch to complete or if this was doing some more extensive number crunching then performing this map operation would actually take quite a while right like if I run this right now it's it's instantaneous but if we needed to do something more complicated here this would actually take longer and just to simulate that I'm gonna make this this transform function just a little bit slower so we're just gonna insert a time sleep call here and we're just gonna sleep for a second and if I run this now I can see here this actually takes a little bit longer to process you know we're still waiting for the results and we can make this some can make this a little bit more interesting because I wanted to be a little bit more verbose I want to be able to see what's going on so I'm just gonna say processing record next name and that's going to give us some output as this processing is happening and then I can say you know done processing or record X dot name and then we're going to return as a result so you know I've made this this function here a little bit more complex and also added some logging statements so that we can see what's going on and now I can run this and I can trace how the data is being processed right so basically this is now telling me how it is running this this transformation and I can see exactly what's going on and I can see how these records are being processed in parallel now up until now you may be you may be wondering if you why why are we doing this right like this is kind of obvious like why why do I even care about this so the cool thing is because we wrote our code in a functional programming style we can actually parallelize it very easily because there is a parallel map construct we can use and that way we can run these calculations these processing steps that now happen sequentially meaning one after another on a single single CPU core we can paralyze them and run many of them in parallel and now that we have all of this stuff in place this is very very easy to do in Python and this is what I'm going to show you in this video so what I'm going to do next is I'm gonna import in the multi processing module because that has all of the building blocks that we need to actually run this operation in parallel and before we move on I want to extend this testbed a little bit more because I want to add some more logging so we can actually trace how long it took to calculate this result so what I'm gonna do here I'm gonna take the start time before we apply the map operation and then I'm also going to measure the end time and what time that time does it just gives you a second space time stamp as a float all right so what we're gonna do here is we're gonna print the time to completion and we're just gonna calculate that as end minus start and that's going to give me the float in seconds that it took to run this piece of code here so now when I when I save this we can run this again it's gonna very slowly process all of these records one by one we're simulating here that this would take up to one second and then it's printing out okay this took seven seconds and you know a little bit more which makes sense because we have seven records in here and I'm just gonna polish that a little bit more just to make sure we have this nicely for minute so I can use a format string here and then run this again do our timing so now we get the input data we get the processing as it happens you notice is logging some stuff and it tells us hey took this long to calculate the result and here it is in seconds you know make it's a little bit more nice just you know limit it to to two decimals and just have a really nice output and I often like to build stuff like that if I'm experimenting with something and it really helps me get what I want from these experiments and from these analyses alright so now we've got everything we need here we've got the strictly sequential implementation of this some this this program here we already also imported the multi processing library so what I'm going to do now we're going to replace the sequential step here we're going to replace it with some multi processing code so what we need to do here first is we need to create a multi processing dot pool object now we need to store that somewhere and so a multi processing pool it's it's basically an interface that we can use to run our transformation or our transformed function on this input data in parallel spread out across multiple CPU cores so this pool instance it has a map function and I can say ok well we're gonna transfer we're going to map the the transform function over the scientists and this is our result so this corresponds exactly to the sequential map function call here I'm just gonna clean it up a little bit and maybe bump up the font size again for you to see and now if we run this what do you think is going to happen so remember before this took about seven seconds to execute and if you run this again now well we're getting a way different output right looks like we're actually starting the processing here for four records all at once and then they all complete as a batch of four and we have another three I guess that's the remaining ones and then we those complete as well we get the same result but we get it a lot faster previously it took us seven seconds now we did it in two seconds that happens because well we have these two batches here essentially now what what is going on here so this is the magic of the multi processing pool because what what it does is it actually fans out it actually creates multiple Python processes in the background and it's gonna spread out this computation for us across these different CPU cores so they're all gonna happen in parallel and we don't have to do anything the result we get is exactly the same the multi processing pool fans out and does all these computations for us applies to transform function and then brings back the results and assembles this output data structure here so we get exactly the same result here which is pretty cool now there's a couple more parameters we can tweak here and I really want to make sure that you see what's going on behind the scenes here so the first thing we're gonna do is we're gonna add some more logging to this transform function so we can see what's going on behind the scenes and for that I'm gonna import the OS module and then here we can print out the process number or some identifier for the current process and we can do that with the OS dot get PID function so we can say process you know the name of the process is working on this record and then we can say process get process ID done processing record and now if I run this again we're gonna get some more extensive output here so each process that happens in parallel it gets a unique identification number and we can see here how how multi processing how the multi processing pool spreads out our mutations across multiple cores so what you can see here is that we have these four processes you know 17 3 24 to 27 and they're working on stuff in parallel and then they're being reused so in the second batch the same processes again are working on other data and we can influence this behavior so we can actually put a limit and say well I only want one process in this pool here and when I run this again you can see here well now we have a single process that's doing all of the computations and again we're gonna end up with a seven-second run time and if you look at the the log here you can see exactly that's the same process processing each record one by one and there's no parallel computation happening and now if I crank this up I can say well we want two processes now if we run this again we have two processes working in parallel and they're they're processing these these records in parallel and now we get a little bit of a speed-up and of course I can go crazy here and I can actually say okay I want a process for each record here so I want the number of processes should be the number of records in this thing here and that way we can process all of them in parallel and we can really cut down our time to complete to a second which is about as long as it should take to process a single element and you know of course this is a bit of an academic example or just a toy example because while we're just sleeping here so there's really you know nothing very interesting going on but if you imagine this was a call that was waiting on IO or it was waiting for a website download to complete if this was a web scraper of sorts you could do all these things in parallel and with multi with a multi processing pool you can really influence how these operations are performed across multiple CPUs or multiple CPU cores and you have a lot of flexibility and configuring it there there's other option here for example we could say okay I want to processes and there's another setting that's called max tasks per child and I can say well I want to processes in parallel and I want the process to restart after it has completed a number of tasks so if you run this we're gonna get a slightly different output here so again if we look at the process ID you can see here okay we're starting out with oh three oh four and those are doing some processing and then we're getting two brand-new processes that are processing the next elements and then we're getting two brand-new processes again and so on so with the multi processing pool you can really influence how it's distributing your calculations across multiple CPU cores and of course you can also leave out these settings and what it's going to do then it's gonna spawn as many processes as you have CPU cores so I'm a dual core with hyper-threading machine that Python sees as a quad core CPU and that means it's gonna spawn four separate processes to maximize the the CPU that I have available and I just love this way of parallel programming because it is very easy to do if you write your code and a functional programming style and if you have if you're using a map function and it's very easy to parallelize this code as you've seen here right I didn't change anything here really I mean we made some cosmetic changes just to be able to trace what's going on with this transform function but really all I did was change these two lines of code and now all of a sudden these calculations are running in parallel across multiple CPU cores and I think that's a really powerful concept now there's more to this of course this is more you know this is really just scratching the surface but I hope it is enough for you to to see the value that this programming model brings with it and to incur like I really want to encourage you with this video to go out and do your own experimentation right maybe turn this into a little web screen maybe you can do some some IO or some more you know more expensive computations here and then use the same technique to to speed this up and actually get your result a lot faster you know if you imagine you had to fetch ten thousand websites well if you did that sequentially this could take a really long time but if you can paralyze them and fetch these websites and batches of 100 items at a time then you can you're gonna get a huge speed-up it will be relatively easy to do this with the multi processing pool a module that's built into Python now like I said there's much more to talk about what I'm gonna do in a future video I'm going to talk about the concurrent module that's built into Python 3 and that all gives us some more flexibility so you can use it again to do parallel programming using multiple processes and then we're also gonna talk about threat pools and what the difference is between the two so yeah give this a shot play with it get familiar with it and then you'll be ready for the next video in this series all right if you enjoyed this video and then click the subscribe button below and I'll talk to you soon

Channel: Real Python

Views: 50,927

Keywords: python, dbader, functional programming, multiprocessing, python parallel, python multithreaded, python fp, functools, functional, functional python, python tutorial, concurrency, parallel computing, python multitasking, multithreading, parallel processing, multiprocessing.Pool, python programming, learn python, python 3, python trick, python map, computer science, lambda, parallel programming, immutability, coding, reduce, filter

Id: aysceqdGFw8

Length: 16min 44sec (1004 seconds)

Published: Wed Oct 04 2017