Understanding Python: Asyncio

[Music] welcome back to understanding python my name is Jake and today we'll continue our mini-series on concurrency by introducing asynchronous programming async can be challenging to get into but by the end of this video you're not only be able to use async you'll also be able to mix in synchronous code as well if you haven't watched my videos on threading and multi-processing I highly recommend you watch them first if you have then let's get started we'll start today by modifying our intro script that we used for our threading video in that script we had two functions that printed names and ages of different people in order to show the time between them we had them sleep a random amount of time there we created the two threads for them we started the threads which they then executed in and then waited for them to complete with join but that's not going to work for async because async Works in a much different way than threading or multi-processing my analogy for threading and multi-processing was thinking about them like a single Lane Road splitting into multiple Lanes allowing faster cars to move around slower cars well the way I like to think about asynchronous programming is more like a fast food restaurant each customer waits in line for their turn to order they place it then step aside to wait for their order to be ready while they wait the cashier can help the next customer the cashier doesn't need to wait for the first customer's order to be ready before taking the next order additionally customers may not be served in the order that they arrived if a customer's order takes a long time to prepare then simpler orders may be ready first so when you're thinking about how async execution works it may be helpful to think back to a fast food restaurant so with that in mind what's it going to take to convert this script to used async well the first thing we're going to do is we're going to create a loop so we'll call Loop is equal to async IO dot get event Loop and of course at this point we need to import async i o from the standard Library so add that now and at the end we know we're going to want to close that Loop so this is effectively Opening Our restaurant and this is closing a restaurant once all the customers have been served and in between these statements is where we need to let our customers in take their order and serve them so how are we going to do that well we're not going to do it by creating threats that's for sure instead we're going to create a list of tasks and those tasks are going to be print names and print age but we can't just add any functions to this Loop we need to add things that are async compatible in other words we need Cooperative customers ones that are going to place their order and step aside we can't do that with these functions however with some minor modifications we can make these Cooperative in order to do this we're going to introduce a couple new keywords we haven't seen in this series before and that is async Def so this is defining an async function print names and there's one additional modification we're going to need here because time.sleep isn't Cooperative time.sleep is a gravy function so it's not going to step aside instead we're going to use async IO dot sleep this is a special function in that when it sleeps it steps aside and the way that we're going to tell print names that async io.sleep is able to step aside is we use the new oh wait keyword so this reads await async IO dot sleep at this point when print names hits this line it's going to trigger asyncio.sleep and it's going to step aside we're going to do the same thing for print age and because the name was bugging me in the last video we're also going to make this plural so finally we have async def print ages and we're going to do the same thing here instead of time.sleep we're going to await async io.sleep okay now we have two async functions defined here now what you'll notice about these async functions is that they have a mix of normal code as well as asynchronous code what that means is that not everything in an async function has to be asynchronous in fact many things won't be really only the things where you're waiting for some type of input or output like you would with threading should be awaited that's the i in the O in async i o async input output okay we have our two functions here how do we add them to the tasks well we're going to do something a bit different instead of passing the functions themselves we're actually going to call them here at this point you may be thinking if we're calling them here wouldn't that just execute them like they would in normal function but the answer to that is no when you call an async function you're actually getting something different back so let's take a minute to pause I'll comment these out we'll save I'll activate my virtual environment and we'll load up this file in IPython to see exactly what's going on at this point okay so we have print names so far everything looks fine it's a function same thing print ages again another function but if we look in the tasks here we see we have co-routine object print names and co-routine object print ages that's because when you call these functions it actually creates co-routines and co-routines are what we're going to pass into the event Loop those co-routines are the customers co-routines are special construct in Python that allow for Cooperative multitasking they're the things that enable us to await different actions yielding back execution to the primary loop we're going to dig a lot more into co-routines into my next video but I don't want us to get bogged down too much into Co routine specifics in this one as we have a lot more to cover so for now we'll exit IPython which it doesn't like because we created a couple routines that were never awaited clear that out and give us our room back okay at this point we have our tasks which is a list of co-routines but how do we execute them well instead of starting and joining these co-routines separately what we're going to do instead is we're going to call Loop dot run until complete so this is what's going to take the orders and fulfill them until all customers have been served but how do we get the customers into the line we do that with async IO dot gather what we're going to gather is an unpacked list of tasks or customers so asyncio.gather is what gets everybody into the line run.complete is what actually takes the orders and fulfills them so this is where the actual execution of print names and print ages happens okay let's clean up our Imports because we no longer need time or threading save it and give ourselves some room run it and there we go we see John a bunch of Ages go by then finally Kate Mike Alex and Anne and since this is random we should see a different order happen this time with Kate appearing a little bit higher in the results okay we just converted a Threading script into an async i o script by creating our first async functions which as we know when we call them create Co routines co-routines are what runs in our Loop which we got earlier in the script we got all of our customers together into the line and then random with run until complete and then finally we manually close that Loop and then why this script may have been helpful for introducing the basic concepts of async I'm personally not a fan of tutorials that just leave it at sleep examples so let's dig into something more real world and we'll do that with our site status example this is another scenario that's coming from the threading video this time we're going to do it with async one thing that we will need is a third-party dependency called AIO HTTP or async IO HTTP which is great library for interacting with HTTP requests in order to install that we're going to use pip so make sure that you're in your virtual environment around pip install AIO http I'm going to spend a little time introducing what's happening in this git status function but this is really only the surface of what you can do with AIO HTTP but there's some important things to keep in mind because it also introduces a new statement for us in that new statement is async with this is an asynchronous context manager and we're going to use two of them the first one we're going to use for client session because this Library manages sessions in a connection pool so much like you could do with a thread pool or a process pull executor this also has a pool of sessions that it can pass executions back and forth between next up we're using async with for the response itself you may have noticed that we're not explicitly doing any awaiting within the git status function well the awaiting is actually happening here with this response so it's going to await the response and another really nice thing is that after we exit the context manager it's going to clean up that connection same thing with the session when we exit the context manager it's going to clean up the session okay so the flow of this function is we're going to get our start time because we're going to see how much time this is going to take we're going to create our session we're going to run our request and await the response coming back in then finally we're going to get our stop time and calculate the total time it took and print that out then return the URL as well as the status okay now execution for this is going to be a little bit different than what we did in our previous example instead of creating a loop let's define a main function first thing we're going to do in our main function is we're going to create the tasks that need to run so we'll simply iterate for each URL in the URLs now instead of just appending git status passing in the URL creating a co-routine what we're instead going to do is we're going to create a task out of this so we're gonna wrap this co-routine in asyncio dot create task the task is effectively a wrapper object around co-routines to help async IO to manage the co-routine itself attached job is to help monitor the execution of Co routines and allow async IO to get the status and cancel but if need be and it's this task that we're going to append to our tasks list okay so at this point we have a list of tasks but how we're going to run them well the async io comes with another nice feature and we're going to use that now so we're going to iterate for each task in asyncio dot as completed and we're going to pass in the list of tasks to that so as completed is going to do is it's going to put all those tasks on the loop and check on them and as one completes it's going to pop it off and give it to us here and now that the task is saying that the co routine it wraps is completed we can pull the URL and the code from that co-routine by awaiting the task itself remember that forget status we're returning both the URL and the response status so that's what we're getting here then we can go ahead and print that out now as a good practice because this can give us an error we'll go ahead and wrap it in a try and accept and we'll just print that the task and the arrow okay now we have this async main function but if we run the script right now nothing's going to happen so how do we run this async function easily enough for functions like this async IO comes with asynqueo DOT run I'm going to pass in a call to main so remember calling this main function creates the co routine itself and the co routine is what's going to do the work okay we have our main function defined which creates all the tasks the tasks are wrapping the code routines monitoring them and making it easier for asanko to manage them we're then going to process all the tasks and pull out them as they complete we get the results and print that out let's give ourselves some room and run this well looks like we have an issue and looking up in my code I see up here it is instead of appending the single task I appended all of the tasks which doesn't really work for those that have been screaming at the screen or already starting to type your comment you can relax now so we'll save and run it again there we go that looks much better okay so we start the script it says that it's getting the status of each of the sites Google comes in first then LinkedIn came in even though it started last Facebook is next GitHub and then Twitter came in last year if we run it again we may have a little bit different result here we go this time the orders Google LinkedIn Facebook Twitter and GitHub but just like that we were able to asynchronously get the status of all five sites with the help of AIO HTTP now this environment works really well when everything is async compatible but what happens if you have code that can't be converted to async meaning you want to mix synchronous code in with asynchronous code well in our last example we're going to take this script which uses sub process to run a series of commands subprocess.run is a blocking command so it isn't async compatible in its current form meaning that each of these commands that we're going to run has to wait for the previous to complete if we do that now we received that who am I runs really fast being in Google five times took a relatively larger amount of time and then the following commands were much faster running LSP release as well as the python version so how do we make this async friendly well our main target here is going to be that sub-process.run command so let's start the conversion first we'll import async i o next we'll convert run command into an async function allowing it to be executed as a CO routine the next thing we're going to do is we're going to use a really nice helper function that async AO comes with on subprocess.run and that is async io.2 thread which is awaitable so async io.2 thread is going to spin up a new thread to run the sub process in now the arguments for this isn't the execution of subprocess.run instead like we did with threading we're going to pass in the target function and then its arguments so the arguments for that being the command itself are pipe for standard in and our pipe for standard out and just like that we made run command async friendly and now that it's async friendly we can't run it like we normally would because remember calling run command here would just create a series of Co routines it wouldn't actually execute anything so let's extend our conversion here as well we'll create another main function we'll pull commands under that and we'll also keep track of a list of tasks move our for Loop over as well and take the co routine that's created here and wrap it in a task then we'll append this time our singular task to the tasks list now unlike our site status example we're not actually returning anything from run command so we can do execution a bit more simply with that so instead of using as completed we'll just await async IO dot gather unpacking our list of tasks into it so this will add it all to the loop and will await the execution of all of those finally we'll call async AO dot run and call Main and pass it into run all right so at this point our conversion should be complete so let's give it a go oh and there we go we have who am I running first it tells us that it's kicking off all the commands one after the other who am I ping LSB release and python version the first to complete unsurprisingly is who am I telling us that the output is Jake python version completed next telling us that we're in Python 3.11.4 next up was LSB release telling us that we are on fedor38 and then finally pinging Google five times came in last because it's the only one that actually reached out to the network and just select that we converted a synchronous script into an asynchronous script if you had more things to do in here say combine it with site status you could absolutely do that as well now that run command is async friendly thanks to asyncio.2 thread it can be incorporated into other asynchronous code now there's a number of different ways we could have done this and there's a lot more to async Il but this is a great place to get started and that wraps up this video now that you have an understanding of how to do asynchronous programming give it a try in a project of your own and let me know how it goes if you have any further questions or recommendations for others leave a comment down below as always today's code will be added to the understanding GitHub repo so check the description for a link and of course if you have any questions or suggestions for topics you'd like me to cover let me know in the comments section to keep up with the series please consider subscribing thanks for watching [Music]

Channel: Jake Callahan

Views: 2,718

Keywords: python, asyncio, python asyncio, python asyncio tutorial, asyncio python, async, asyncio python tutorial, async python, python async await, python await, python programming, async await python, python async, asynchronous programming python, python asyncio vs threading, asyncio in python, python asyncio gather, asyncio python api, python asyncio explained, asyncio python 3.7, python asyncio example, python async vs threading, async in python, understanding python

Id: ypumJY4zFMw

Length: 19min 43sec (1183 seconds)

Published: Mon Jul 31 2023