Python Asyncio, Requests, Aiohttp | Make faster API Calls

so there i was trying to download 16 million colors using the request library in python all right here we go all the way to 16 million six and a half hours later oh okay at this rate i'd only have to wait 66 days and we'd be all done trash obviously nobody wants to wait 66 days to have to run some simple script this is where asynchronous programming lends a very helpful hand and it's what we're gonna be showing you how to do in python let's look at an example there's a link in the description to all the code that we're gonna be working with here i have two scripts here that for all intents and purposes do the exact same thing they both make 100 api calls to the alpha vantage financial data api we're looking to get a ton of financial data very quickly it is also going to time how long it takes to make 100 api calls let's run each of them a few inches later 20 seconds that's a snooze fest amount of time let's try the other one now let's try async one second how we need to talk about asynchronous programming and the event loop an event loop is a loop that can register attacks to be executed execute them to play or even cancel them blah blah blah blah blah blah what the hell does that mean it means we can let some stuff that takes a long time to do its thing in the background and when it's all set the event loop will actually pick it up what do you mean that wasn't better so confused don't worry let's look at a synchronous versus asynchronous example using cooking as the analogy instead of programming this guy is your python code he needs to cook dinner which consists of burger and green beans these functions cook burger and cook green beans are analogous to making api calls or doing some type of input output operation because we need to wait for them to do something we need to wait for the api calls to be returned similarly we need to wait for the burgers and the green beans to cook now he's also going to need a couple other functions including getting the burger out getting the beans and setting the plate this is the entire set of functions that he needs to actually cook dinner get burger get beans cook burger cook green beans and set the plate in our first example he'll be cooking dinner synchronously this is just like python first we call get burger get the burger then we call get beans and get the beans now we start our cookburger function and since we're doing this synchronously we just have to wait for this function to complete in order to move on since this function is blocking us from actually continuing to move on the next function you'll also hear this referred to as blocking this doesn't take any computational power on our side to do we just have to wait so hypothetically we could be doing other stuff like dancing or like cooking the green beans which is what we should be doing but nope we're watching the burger it's only once the burger's done can we move on to actually cooking the green beans making us take a lot longer than we probably should yeah throw them in the microwave this isn't programming with chef ramsay and then when it's finally done we can call our set plate function having dinner done in about 20 minutes so how do we improve this well ideally we see two areas where we can actually do work at the same time during cooking the burger and cooking the green beans and this is where asynchronous programming and the event loop is actually going to come into play we can start an event loop which means we can periodically check on stuff that's already running we can kick off a function and not have to wait for it to complete but periodically check back to see if it's done this is known as the event loop this way we can start the burger and move on and start the green beans but periodically check to see if the burger is done and also periodically see if the green beans are done this way we can get our delicious food done quicker now let's see it in python synchronous first hopefully you'll recognize a lot of this we're going to import the request library so we can make some api calls we're also going to import os just to get some environment variables if this confuses you don't worry about it so i'm going to get my api key os.get env i've already set it as environment variable we're going to be making api calls from the alpha vantage api what do these do well they can give us financial data about stocks and stuff i'm gonna get my url which i copy paste from the alphavantage site and i just format it so that i can add a symbol an api key then i create a list of symbols that i want to grab from maybe i want to get apple and in google and tesla and microsoft and i don't know let's just go with four for now i'm going to return all the results here now all i got to do is loop through the symbols so for each symbol in symbols response equals requests.get url.format symbol api key results.append response.json print you did it print working on symbol this dot format symbol you can zoom out here real quick we are synchronously looping through all the symbols in here and just printing out that we're calling that api so let's see let's run this amazing that was so fast we can even time this do a little bit of fun timing here wow it took 1 seconds to make 4 api calls you did it great job but what if i want to make a lot of api calls how long does this take 13 seconds to make 60 api calls huh i can get through two tick tocks in that time so this is clearly too long so we want to be able to do this even faster let's do this exact same thing but asynchronously so let's even start by copying this whole thing and pasting it in here so this is going to be exactly the request but we want to do this asynchronously so the main things that we have to import are going to be asyncio which is the library that allows us to actually run stuff with event loops and run subsynchronously because python natively does not do that and aio http since the request module is actually naturally synchronous so we want an asynchronous way to make these api calls everything in here can stay exactly the same we'll keep the 60 symbols now let's come down here and we can even comment out all this timing part because that might just be confusing this is the code that we care about and this does it synchronously so how do we do this not so synchronously well at some point we're going to need to get that event loop first thing that we're going to need to do is we're actually going to need to tell python that this block of code is asynchronous and then we can use asynchronous stuff in it so we can actually turn this whole thing into a function async def get symbols and now this keyword async tells python that this function can now do asynchronous stuff synchronous asynchronous synchronous asynchronous get symbols and let's just have a function do get symbols and if you run this right now you get this error saying get symbols was never awaited so we need this event loop right the event loop is what's going to be checking to see if there are async functions that are done or async processes that are done so we need to start that event loop or that checker so we can do event loop equals asyncio.get event loop and this is going to be our event loop so we have some loop that's going to be checking stuff and we want to run this get symbols thing inside of the event loop so we'll do loop dot run until complete get symbols and this is saying that getsymbols is going to run inside this event loop and then after everything is done get symbols if we run it now it's going to work exactly the same right so it's still going to take a really long time to do this because we're still using requests so let's come and pick out the timing stuff so we're running this stuff in a loop and this this is the function that we're currently running inside of a loop but nothing in here is asynchronous this is all synchronous stuff too get rid of goodbye requests now the other thing that's kind of confusing is that this is like hey there's a lot of stuff here but all we're really doing is running this get symbols luckily async gives us a way to actually abstract all this out we can just do this by calling run async yo dot run get symbols and it's the same thing as doing all this loop stuff here but we just say async you dot run so this is the same as doing this but this is only one line and in python we like doing stuff that's easy and in one line so great so now we got rid of requests let's figure out how to do this stuff asynchronously now so we're running an event loop but we're just not doing anything with it we're just doing synchronous stuff inside of this event loop let's do asynchronous stuff inside this event loop so instead of this requests thing we're going to use aiohtp so we could do session equals aiohtp dot client session and this will get give us a session that we can make these get requests with but because at some point we're gonna have to do session.close and sometimes and by sometimes i mean all the time i forgot to do session.close it's usually a better practice to actually do this syntax which is going to be with aio http dot client session as session excuse me client session with parentheses as session and pop this out this means that whenever this finishes it's actually going to automatically close a client session however if we run this right now it's going to say unclose client session use async with instead so we want to use this async width instead of just with we're saying hey this is an async thing and then of course now if we run it it just says hey there's no such thing as requests well great we don't care this means that we'll run this bit asynchronously so this says this function is async this says this width is a async which is great that's what we want now the code to make the api call is just this instead of requests.get we're just going to do session.get so inside of this session that we've opened up with the internet we're going to get this url and we're going to format it the same way we did the request one other thing is oftentimes you'll have to do ssl equals false here now here's the other thing when we make this call this is actually not going to return the response why this is not going to return a response like it does with requests it's actually just going to return a co-routine so this session.get and is is an example of a co-routine what is a co-routine and in order for a response to actually get saved someplace we need to wait for this to make the call and return so if we want to do that we can do await session.geturl.format so we're waiting for this to return otherwise this will return nothing it won't start it won't do anything so we're saying hey we're going to go ahead and kick this off throw it onto the event loop and wait for it and then we can do this results.append response.json which we also have to await because this is an asynchronous awaitable object so now when we run this function it's going to go a lot quicker because these are being sent asynchronously however the waiting and and let's even add the time in so i guess nine seconds so about four seconds quicker it doesn't feel like it's that much quicker right well you're right this isn't that much quicker the reason being is we're actually waiting for the url to get sent and responded to and then we're waiting to get the answer here so we're really not doing that much better than with requests we are kicking them off a little faster in this event loop but really not doing anything that much quicker but for all intents and purposes this is some async code great job now let's spice this up let's spice this up so we have this code and we're waiting to get a response here we're waiting for session.get to return and we're doing this for every single of these it's not really the same of just throwing the burger on the grill and then walking away because we're actually waiting to get a response here similar to synchronous we want to send the api call and not wait for the respond and we want to just rapid fire send all the api calls so how can we just send all of these at the same time well this is where we can get into tasks so we can create a list of tasks for the event loop to use and just send that entire list of tasks at once so instead of doing all this hard work here what we're going to do is we're going to create this function and it can be synchronous called get tasks and pass in the session in this we're going to create a list of tasks which include everything that we want to run in this event loop so for each symbol in symbols tasks append session dot get and it's going to be this this bit here boom and then we're going to return return the tasks now instead of doing this stuff we're going to say tasks equals get tasks with the session that was passed in we're going to do now is we're going to tasks is get tasks from the session this is a list of all the functions that we want to call so right so it's test.append session.get so it's all these functions that we're going to call we're going to take the entire list and we're going to say hey let's kick all of these off go have fun do it and we're going to do that by saying responses equals await asyncio gather star tasks what does the star do well it dereferences the list so it's going to do session.get you know the the tesla api call comma session dot get the microsoft api call you know etc etc etc but it's all packed up very nicely like this so the star dvr references the list and turns it into unique parameters for this async eo.gather so we're going to send everything to the event loop and then we're going to do for each response in responses responses we do result results dot append response and we actually have to do results.append await response.json because response is an awaitable object so again what we just did here is we got all the tasks packed together in a nice little list we threw all of them onto the event loop with this await asyncio.gatheredtasks and then once they're done we're just going to grab them because we're going to wait for all of the tasks to be done here so let's see how this performs now oh my goodness gracious it took less than a second to make all 60 of those calls you did it great job now i'm going to show you one other advantage here so if you want to just throw these tasks on the event loop right away you can do this thing called asyncio.createtask session.url this session.get is a co-routine it's an awaitable co-routine but it's not put on the event loop if we do asyncio.createtask it's automatically thrown on the event loop right here then when we call await asyncio.gathertasks this won't actually throw them onto the event loop this will just wait for them all to come back this list of tasks can be stuff that's already on the event loop or stuff that needs to go on the event loop and a cinque duct gather it'll put stuff on the event loop if it's not already and then we wait it for it to come back all the code that we want over here is going to be in the description have fun talk soon chapter reveal at 10 000 you

Channel: Patrick Collins

Views: 9,872

Rating: 4.9000001 out of 5

Keywords:

Id: nFn4_nA_yk8

Length: 17min 56sec (1076 seconds)

Published: Tue Apr 20 2021