Python Asynchronous Programming - AsyncIO & Async/Await

[Music] hello everybody and welcome to another youtube video so in today's video i'm going to be discussing asynchronous programming in python specifically what i'll be covering is the async and a weight keyword co routines futures and tasks and some basic features or functionality from the async io module in python now it's worth noting that this video is not designed for beginners you should have some basic programming knowledge in order to follow along especially in python and if you're using any version of python that's below 3.7 a lot of the stuff i'm showing you here will not apply so please make sure using python 3.7 version or above and with that said let's dive in after a quick word from our sponsor before we get started i need to thank the sponsor of this video which is alco expert algo expert is the best platform to use from preparing for your software engineering coding interviews and has the highest quality coding interview practice questions with over 140 practice questions detailed solutions in nine of the most popular programming languages a feature-packed browser-based coding environment extensive test suites and conceptual overviews and code walk-throughs for each and every problem algo expert is the best resource to use to ace your coding interviews algo expert also has a data structures crash course coding interview assessments and a mock interviews feature i can highly recommend algo expert as a former customer myself and now an official instructor on the platform get started using algo expert today by clicking the link in the description and using the code tech with tim for a discount on the platform so before we get into asynchronous programming i want to start by defining what synchronous programming is so synchronous programming really just means that everything that we write is happening sequentially so if i define some statement all right to find some functions sorry and i call this foo so define foo and maybe in here we just do a return statement then i call foo and then i say print and i print tim this print statement is not going to be run until this function foo is executed doesn't matter how long this function takes doesn't matter if this function's actually doing anything or not this function must finish executing before this statement is going to be run this is by definition synchronous programming we are doing things sequentially and i like to think about synchronous programming as in having a kind of performance based on the clock speed of our processor so when we're talking about a python program specifically we have one process and one thread running our code which is just standard if you don't know what that means don't worry about it that's just your regular python code running there well the speed at which this code is going to run is going to be dictated by your processor the state of the processor and how fast essentially it can actually run how quickly or how many operations it can do per second so if you have a fast processor your sequential synchronous code is going to run faster than a slower processor however when we go over to asynchronous programming things are a little bit different when we talk about asynchronous programming we don't necessarily need to do things sequentially and the reason for that is the following let's say this function foo here this function foo actually waits on something that's not related to our computer so it's waiting on say a network operation waiting on some file to upload or download or some user to uh input some information while that function is waiting other code is able to be ran so we don't need to do things sequentially we don't need to wait for foo to finish running before we can print tim we can say okay well foo's waiting for something else to happen our processor is free to do some operations so let's allow us to go run some other code and then when foo is done we'll come back to it later on and we'll kind of check on it or resume the execution of it now i understand this is maybe unclear and vague right now but you'll see the idea behind this when we actually start writing our asynchronous code so just understand that when we're talking about synchronous vs asynchronous programming asynchronous programming really means that we don't need to necessarily wait for something to be completely done before we run other parts of code and this is very beneficial especially if you're waiting for say a network operation like you're querying a database you're waiting for a response from some server that's not disallowing your computer from running you can now execute other operations then once that response actually comes in or once that event you're waiting for happens you can then handle it and do something with it so with that said let's dive in the first thing that i need to discuss here is something called co routines and the await and async keyword okay so as you can see here i've imported this module here it's called async io now this module is built into python you don't need to install it and you can import it right here if you're using a version of python that's high enough again i'm recommending 3.7 and above now the first thing that we need to do when we're creating an asynchronous program is we need to create what's known as a co-routine now a co-routine is simply kind of a wrapped version of a function that allows it to run asynchronously because you have to think about it in this way when we're writing code traditionally we're writing synchronous code to be able to write asynchronous code we obviously have to do things a little bit different we can't use the exact same code that we used when we were writing something synchronously and so to create a code routine what you do is you define a function so i'm going to say define main and inside of this function you do whatever you want so i can print say tim and then maybe i can sleep for a certain amount of time or whatever i'm just going to print him for now and then before this function definition you put the keyword async now what this tells python to do is essentially create a wrapper around this function so when you call this function what it's going to do is actually return to you a coroutine object and this co-routine object is just like a function and it can be executed and to execute a code routine you need to do what's known as await it you need to await its execution now you'll see what i mean in one second but let me just show you how we kind of work with code routines and what a curve routine is so right now if i call this main function you would expect that tim's going to be printed to the screen so let's go ahead and do that and notice we get this thing runtime warning co-routine main was never awaited now this is telling us that hey we can't run this function because what you've done is you haven't awaited the code routine and when you call this function well it actually returns to you a co routine it doesn't work like a regular vanilla python function would now let me show you what i mean here if i print the call to main you're going to see up here that actually gives me what's called a coroutine object it says well this is a coroutine object and it's main at and then whatever this memory location is so calling this function doesn't just execute its code it now gives me this kind of new function or new object and this new object is asynchronous and can be used in the asynchronous program so if we want this to actually run what we need to do is what's called awaited although you're going to see what happens when i do this so when i await main we actually get this issue and it says there's an await outside of the function now this is because to use the await keyword it must be inside of an asynchronous function now we'll talk more about that later i don't want to get too far ahead of myself but essentially what i'm trying to say here is that when you call a function that you've defined with async it doesn't work in the way that you would kind of imagine and intuitively see in regular python instead what this returns to use that coroutine object and that coroutine object now needs to be awaited for it to be able to run but you're seeing this error right here i put this a weight keyword which you've probably seen if you've looked at asynchronous programming before and well it's not running our code why isn't it running our code the reason it's not running our code is we haven't created what's known as the event loop now whenever we create an asynchronous program at least in python here we need to define something called an event loop where we need to at least start an event loop now an events loop is just kind of the lower level python thing that's allowing us to actually write asynchronous code it's handling all of the super complicated stuff that actually is behind this simple async syntax we see right here and we must start one in whatever thread we're running this asynchronous stuff in before our code can run if you're unfamiliar with reds don't worry about it just mention that in case someone uh you know knows what a thread is and that can be helpful anyways the whole point of me saying this is that we need to start our program so whatever asynchronous part of our program we want to run we need to start it with async io dot run so we say async.oh async io dot run sorry we then give a co routine and this co routine is going to be the entry point to our program again a co routine is any function that has the async before it so now when i do this you're going to see that tim runs no problem the reason that runs no problem is this is the code routine we told async io dot async i o to run it async io created what's known as an events loop it added this co routine to the event loop and then it ran it that's all you need to understand now you don't need to know too much about the event loop just know that really what this is doing is creating the event loop adding this onto it and that's what's allowing us to actually run this code so now let me show you how we can actually use the await keyword so i'm going to create another function here i'm going to say async define and let's just call this let's say foo now inside of here what i'm going to do is just print out some text and actually let's take some text here and we'll just print out the text and then i'm going to await a function i'm going to await something happening i'm going to say await and then this is async io dot sleep and i'm gonna sleep for one second now this await keyword right here this is what is required to run a co routine so async io.sleep one returns to me a co or something like a car routine i won't get into all of the details of that and in order to run a co routine it must be awaited you can't just do it like this the reason you can't do it like that is because you're not actually executing the car routine it's just creating the car routine you need to execute it with the await keyword so i use a weight and the reason i'm allowed to use a weight here is because i'm inside of an asynchronous function so this await keyword can only be inside of something that is defined with async if you don't have async like you saw previously when i did say a weight whatever it just doesn't work because you're not inside of an asynchronous part of your code so you put a weight here that then runs the co routine which means we are going to sleep we're going to pause the execution of this function for one second so what i'm going to do now is call this function foo from inside of my main function and again notice if i just put foo right here we're going to get this error and it's going to say oh missing one required okay let's put text in here first but if i do this it says runtime warning code routine foo was never awaited so now if i await this code routine then this actually works right you see we get tim we get text and then there was kind of a little delay and then the program finished because we were sleeping for one second so that is how you actually run asynchronous code routines now this doesn't seem entirely useful right because again this is kind of just like another synchronous program i'm awaiting some code to happen once it happens i continue on with my program let me show you what i mean if i go down here and now i print finished what this is doing is i'm printing tim i'm awaiting this function to finish running that's really what this means it means execute this code routine and then i'm printing finished so if i do this we're gonna get tim text we're gonna wait one second because we're sleeping we're awaiting this operation happening here and then we're printing finished now remember what i told you though is that the whole point of asynchronous programming is that if this function isn't really doing anything if it's just waiting for some operation ideally we should be able to run something else right we should be able to have our code do something else and that is what i'm going to show you how we can do right now so just to backtrack a little bit in this example right here we printed tim we awaited something it ran and then we printed something else now ideally when this function is sleeping when it's waiting for something to happen we actually want to be able to run this next line of code i don't want this line of code right here to be quote unquote blocked by this line here i want it to be able to run when this isn't doing anything so how do we get that to work well this is where we create something called a task now to create a task what you do is you remove this away keyword right here and i just need to look at my cheat sheet i'm going to say task is equal to and then async.io dot create underscore task and then you put the function or the code routine that you want to run so when i do this when i create this task really what i'm telling python to do or i'm telling async.io to do whatever is to start executing this as soon as it possibly can and then to allow other code to run well this task is kind of stalling well this task is not running itself so this task has a point in time where it's sleeping for one second no operations are going on so while that's happening i should be able to do something inside of this main function now let's just have a look at what happens when i run this so when i run this code i get tim i get finish and then i get text now it's interesting right that why this here ran after this finished print statement i created this task right which tells python hey start running this as soon as you can but finished this print statement executed before this i guess text statement was printed out the reason for that is that when i create this task this main function is still running although i'm telling the program hey i want to run this task here we're still waiting for this main function to run so the reason that this is the case is because when i create this task i'm telling python hey i want to run this task but he cannot run this task until the execution of whatever function or whatever thing it's currently doing is done or until whatever it's doing takes a break and stops you have to understand with async io only one thing can happen at the exact same time it's just when something is not doing something execution can be paused and kind of given back to another function so if i wanted this task to run or i wanted to wait for this task to finish before i run our rand finished what i would have to do is await it so i can say await task and what this now does is it says all right wait until this task is finished that's what this line right here does that's what it says wait for this to finish and only once this task is finished move on to the next line this line of code here will block us from moving any further in the program it's waiting for this task to be finished so if i do this we get tim we get text we wait a little bit and then we get finished because we awaited the task but when i removed that again you saw that we just ran all of the code immediate right so this is kind of the benefit of async io now what's interesting here is that if i decide to take a delay or i decide to add some sleep here into this main function you're going to see how this task kind of behaves so if i go now here and i say async io.sleep and i sleep for let's do two seconds so we can actually really see the difference here you're going to see what happens is that as soon as we hit this word right here we hit this line this task will actually start executing so let's watch this we're going to say async io dot sleep notice we print out text and then we print out finished so because i created this task i'm saying okay i want to run this as soon as i possibly can now inside of this function here which is currently running it sees in a wait we're awaiting something happening in this case we're awaiting uh just asleep which means we're going to delay the program for two seconds so as soon as we see this delay we say okay we're going to pause the execution of this function and we're going to now allow this task that wants to run to start running because if this function is sleeping well there's no point in having the processor just sitting there doing nothing so let's allow our task to run and so what happens is this task now runs right but now let's see what happens if i make this sleep 0.5 seconds so if i say task equals async io dot create task foo text and then i say wait async io dot sleep 0.5 and then i print finished and then i go async.io and let's actually make this sleep 10 seconds just take a guess or kind of you'll see what happens here i'm going to explain it but let's just look at this so say tim text and then finished now notice right that really what's happening here is we're printing tim we have this task since we're pausing here we're going to allow this task to start running and then inside of this task we're waiting for 10 seconds so if we had to wait for this task to completely be finished running before we went back to this main function we would have waited 10 seconds but since we saw this awaiting here what happened was now this task gives the executioner kind of gives the resource back to our other function and now allows this function to start running so as soon as this sleep is done 0.5 seconds we can move on to this next finished line we don't need to wait the 10 seconds for this task right here to be done that's kind of the basics or that's how you're taking advantage of async io when you create these tasks this allows you to run things concurrently which means that you don't need to wait for something to be completely done before you can execute something else okay so i've just cleared the screen and kind of created a little sample program here just to walk us through more in depth how this actually works so i have two programs or two functions sorry two co-routines really i have fetch data and print numbers inside of fetch data i print start fetching i wait two seconds this is to simulate like we sent a request to a server or something and it took maybe two seconds to get a response we then print done fetching then we returned some actual data we returned data and then colon one this is just to simulate that we got some json response or something we then have uh print numbers inside of print numbers what we're doing is we're just printing the numbers from zero to nine and then we are waiting 0.25 seconds every single time we print a number so maybe this could be something like a request coming to our website maybe every 0.25 seconds some request is coming in and then you know we're fetching data that takes us two seconds so what i want to show you here is how we would run both of these at the same time and then i'll compare that to kind of the synchronous version so i'm going to create two tasks here i'm going to say task 1 is equal to async io dot create underscore task i'm going to create the task of and this is going to be fetch data then i'm going to say task 2 is equal to async.io dots sorry not sleep create task and we'll create the task of and this can be print underscore numbers now what i want to do is i want to actually get the value from here right from fetch data so ideally task 1 really should hold this data right because well if this is returning something to me and i've created a task how do i actually get this returned value well this is where there's something called a future so when you create a task and the co-routine that you've defined returns a value this creates a future now a future is like a promise in javascript if you've seen that before all this means is that this is kind of a placeholder for a value that will exist in the future so right now when i start running this task i don't yet have my return statement right i don't this code routine is not finished executing uh it may be paused in the middle i don't know what's going on with the code routine all i know is that i don't immediately have this data i can't immediately return this information and so before i can access this information i need to make sure that this task is finished so if you want to actually get the value returned from a co-routine you must await that co-routine or await that task so in this case what i'm going to say is a wait and then task one and specifically i'm gonna say value is equal to a weight task one so what this will do is ensure that task one is finished and then assign any return value that it had to this variable value so we'll wait until this task is completely done that value will then be assigned to the value variable if you don't do this if you say value is equal to task 1 and then you print out value i'll show you what actually happens here so let's just run this and let's see what we get we get start fetching we get zero and you can see here that the print statement up here said task pending name equals task two fetch data running at some location so when i printed this out that was the very first print statement that we had here well nothing happened and actually now that i'm looking at it too we only printed one number and we didn't even finish doing all of the fetching right and the reason why that's the case is because our entry function here well after these two tasks are scheduled and then we do these two lines it's done we're done executing this co-routine is now finished and well we're done and so we didn't wait for these things to finish and since we didn't wait for these tasks to finish well we didn't get a value and we didn't print all of these numbers so if we want these tasks to finish or we want to say get these values we must await them so i say value equals a weight task 1 and then i print the value now you're going to see what happens when i run this code we're going to print a bunch of values and then as soon as we get that value that we're looking for it says done fetching we ran that then we got the data and we printed it out but notice we didn't finish printing all the numbers there was one more number we had to print nine for us to be finished with print numbers the reason this was the case is we waited for task one to finish so we waited for these two seconds to be done and then as soon as they were done we just ended the program so after this after we get our value we can print that out that's fine but we also need to wait for our other task to finish so we need to say await task two and now if we run this code you're gonna see that we will print out in the same order and then we will print out nine we now have printed out nine because we now awaited task two to finish so hopefully this makes it a little bit more clear gives you like a decent example but what's happening here is we start task one we start task two and then we see this await statement when we see this await statement this tells our program hey we got to wait for task 1 to finish running before we can move any further down in the program and so we wait for task 1 which is fetch data to stop running but in task 1 we delay right we delay two seconds so as soon as this task is delayed this other task that is also a part of our events loop then is allowed to start running so it starts running it delays for 0.25 seconds when it delays it says okay can i give execution back to any other tasks it looks at the other tasks and they're still delayed and it says okay no i i can't give it back to them so it keeps running so that's why if you look at this time increment right we only print oh i can't i mean we can just run this one more time we're only printing done fetching at time eight the reason that's happening at time eight is because once our delay gets to about two seconds this now is done we can give the execution back to this function and then we can print out the value you get our return value all of that so hopefully that's like a decent example and that helps you understand a little bit about what's happening here but if we change this delay to 0.5 you're going to see now that we'll print this much sooner so let's do this and much sooner or not in terms of like actual time but in terms of in relation to the numbers being printed out so now notice that as soon as we get to three we print on fetching right because now since we're delaying 0.5 seconds as soon as we're at about that two second delay mark and as soon as this function can start executing again it starts executing we give the execution back and then this function is now done right so we you know passed this await here and we now finish or we wait for task two to finish and then we're all good and we stop now one last thing here is important to understand that these tasks they subclass something called a future now a future really just means that hey there's a value that's going to be returned or that could be returned in the future that's why it's called a future we may not have that value yet so we need some kind of substitute so that we know that a value is coming and so that's why when i printed out value before awaiting the task you saw that it said this task is still being completed it's not finished yet and then as soon as it was finished i was able to actually grab that value so if you want the value from an asynchronous function you must await it and if you create a task you can await the task to finish which is what we're doing right here whereas if i just await task one so if i say a way to fetch data now and i don't create the task this now is gonna wait for this to fully finish before task two can start running so let's run this and see it says start fetching we're gonna wait for it to end fetch uh oops wait what's the issue name task one not defined uh oh it's because i didn't okay sorry let me just remove these two lines here and run this okay so start fetching right done fetching and then it starts running task number two because we need to wait it for this before we define this task whereas if i now define task two up here and i start running task two and then i weight fetch data you can see what happens so it says start fetching then it will start running task two finish the fetching and then we're all good right and then even down here uh actually that's all i wanted to show there's nothing more that i can go through there okay so i think that's all i wanted to go through with async io i apologize if this video is kind of all over the place hopefully i gave you enough knowledge to start understanding async io and actually using it but understand you can only use the await keyword inside of asynchronous functions so inside of an actual code routine when you create this async function what you're really doing is wrapping another function with an asynchronous version of it that version is the co routine to actually run the code routine you must await it in some way or add it to the event loop adding it to the event loop can be done by creating a task this allows you to run different uh co routines what do you call it concurrently and to start your asynchronous program you must create an event loop that can be created by using asyncio.run and passing an entry point co-routine which in this case is the main function so hopefully that was helpful if you guys enjoyed make sure to leave a like subscribe to the channel i will see you in another youtube video

Channel: Tech With Tim

Views: 344,930

Keywords: tech with tim, synchronous vs asynchronous, synchronous programming, asynchronous programming, coroutines, async event loop, async keywords, await keywords, tasks, async example, python programming, asynchronous programming python, python development, AsyncIO, Await/Async, Python Asynchronous Programming, futures, coroutines in python, event loops python, futures in python, tasks in python, how to do asynchronous programming, asyncio module, await/async keywords

Id: t5Bo1Je9EmE

Length: 25min 57sec (1557 seconds)

Published: Sat Apr 03 2021