Django async HTTP requests with asyncio and aiohttp

hello my name is xander welcome to django core in this tutorial we look at an example of asynchronous http requests with django async io and aio http in this tutorial we focus on using async within the django views so this goes a long way towards illustrating the core concepts of async within django or just async io in general so more specifically this tutorial we're going to showcase the performance of async functions in contrast to the synchronous approaches so we're going to go through a number of different steps starting off with a synchronous approach to gather data from an api collect data from an api and then we're going to slowly move that forward utilizing a synchronous asynchronous approach so at the end of this tutorial we hope you have a better idea of how to implement a some simple asynchronous code in the django view to collect data from an external api if you are completely new to async io and python it's well worth having a look at this tutorial first the beginner plus async i o tutorial this tutorial will take you from the ground up giving you the underpinning information which is going to help you better understand this tutorial the final code for this tutorial is available in the link in the description in the video so please go ahead and have a look at that if you need it now you can apply this code absolutely anywhere it just happens i'm just using the previous tutorials code here where i have a book application here uh so i'm just going to go into the views i've just removed everything from the view i do have a template with an index i utilize that to output some data i'll do that shortly so like i said you can download this tutorial setup using the link in the video description so just before we get to the first example i just want to briefly introduce aio http so this is an async http client server framework now you're probably asking why am i using this so like i said this is an asynchronous python framework now essentially why i'm introducing this in this into this tutorial really is to highlight this tool is a very popular tool that provides some interesting features that we might want to use to support asynchronous programming here with django so it does support client and server websockets middleware and signals so it does have a wide range of tools that i'd be interested to explore further particularly websockets in a later tutorial now this api is designed to make the most out of non-blocking network operations and in here we are showcasing using asynchronous code with network operations so it made sense in my mind to kind of highlight and utilize this tool in this situation so let's go ahead and install it right so that's up then so let's go ahead and store i think i've already installed it okay so we've installed the latest version so let's go ahead now and just import it into our project we're going to be utilizing this shortly of course right so let's just go for a simple example to begin with using async so straight away within django we can use async and then we're going to create a function here and we'll just call this example so the reason why i've called it example uh is simply because i do have a i do have a url already set up for this so that's in my core here url so i've got this path here views example and that would just point me and take me eventually or allow me to show this template right so let's go back into the the views so that's my example sorted so what we're going to do first is and like i said at the start this is a showcase of utilizing or getting data from an api and we're going to focus on looking at performance so we're going to start with a synchronous piece of code and then move across to asynchronous and have a look at the performance difference but first of all let's just set up a simple async request to collect some data from an api so i've gone ahead and pre-written this code and i'm going to bring it in line by line and just explain what's happening so first of all we need to bring in the render so that we can output something to our template of course so let's go ahead now and bring in ao http so we first of all we reference the client session as session so we're going to refer to client session as session uh throughout now the rest of our program so what is client session so in ai http the class this is the class for creating a client session and making requests so this is inside of client session we're going to it's going to enable us to access methods which is going to allow us to get data from an api for example so all the tools we need there to utilize ao in this tutorial is going to be within our client session so let's go ahead now and you can now see we've got another async with session and we're going to be utilizing the get method in order to get the data from this api endpoint so we're going to return the data as res and we're going to access the data through res as we can see here and then we're going to store that in our function sorry enough variable called data so we await that and it returns the data and now the data's in our data point here in our variable data we can go ahead and print that out if we want to or we're just going to now just render that out so let's just render that data out and we're going to be utilizing the index template here and we're just going to pass that in utilizing the data key here so maybe no surprises at this point so let's go ahead now and go into our index page you can see i'm just going to print out the data that we've returned now there's only one line that gets returned here uh one piece of data so let's go ahead now and just access that so if you wanted to actually to know a little bit more about the client reference or in the ai http package framework um but i always go over to the documentation there's great documentation here this takes you through the client session in a little bit more detail so let's go over to our website and refresh you can see that each time i refresh it's going to return a new fact so this is a piece of code we can adapt to any endpoint really so another popular endpoint i see a lot of people who go into interview situations and have to do tasks i find that they are referenced to the pokemon api quite often so let's just bring the pokemon api or one of the pokemon apis in and at the end here we just need to number the number of the pokemon we want to return so for number one for example so this is another api endpoint so we should now be able to just output that see what gets returned there we go so this is a just a reference to in this case the pokemon name overgrow so this is all the data about this particular pokemon so this tutorial is a showcase of the performance gain of utilizing asynchronous code so to showcase that really what we should ought to be doing is collecting more than one piece of data from an api so we're going to collect around about 100 items from this pokemon api i would do this first of all in a synchronous manner and then we test the performance of that just the speed it takes to gather the data and return it and then we'll slowly feed in asynchronous code and start testing it as two separate points to see the performance gains so let's go ahead and import time so we can start to roughly measure how fast our functions are so we're going to need that and we may as well bring him while we're here the async i o tools um ready those and in addition to that we're probably going to need uh let's import requests so we're going to make a request to an api using a synchronous approach and we're going to be using requests for this so this is a package that we need to install so let's just go ahead and pivot install requests so just close this i've already got installed i think there we go so that's going to allow us in ace with a synchronous code to collect data from an api okay so we'll probably utilize this a little bit later so let's just comment that out so now let's go ahead and just create our traditional function our synchronous function and fetch some data from our endpoint so let's start off by importing time we're going to utilize that to time how quick the data is or how slow the data is at returning and then we've got request render http in async io okay so those are all the imports we're going to need here so let's go ahead line by line again so i'm just going to create a new function here called traditional function um synchronous function called example and we go ahead and we start the time so we're going to capture the time at this point and then we capture it at the end and then obviously we just need to do a simple calculation to then work out the the total amount of time that's occurred from the start to actually processing um collecting the data from the api to it finishing right so we get that from time time so we get the time and now we're going to create a simple four uh so we're going to collect the pokemon cards one to a hundred so that's our range here one to a hundred and then for each one of those we're going to send off a request in asynchronous way of course we're going to send off a request to our endpoint with our number from our range obviously that's going to be cycling over 1 to 100 so we're going to get 100 pokemon cards back or data back and then we go ahead and then run that request so we're using requests package here to run that request we're going to get the data return it into res and then what we're going to do is then just prepare that data so we're going to put that into a new list so we basically just use append here to put that data so we're just going to collect the name from the data that's returned we're going to put that in our list here okay so using append and then once we've done that we are then i'm just going to count so i'm just going to make sure that i have received 100 pokemon cards uh so i'm just going to use len to count the the amount of um items here in my list and then i go ahead and finish the time so now what i'm doing is i'm just going to take the time again so the time is taken for the the program to run this code capture all the data and then i'm just going to take it away from the starting time and that's going to give me the total time it took to perform this action here right so once i've done that i'm just need to output that so i've got my render again index.html i'm going to pass over the pokemon data um all the cards that are returned the count so how many how much data has been returned or just uh listing through the the list how many items in the list and then the the time so the total time it took right so this is what the final code looks like now remember this is a synchronous approach here so that means that i'm going to have to send a request individually for each item like i would do an asynchronous of course and this is going to naturally take a much longer time to perform because there's going to be a lot of blocking here in place before i can then perform the next action i can only do one thing here at a time right so now i've got that in place the server's on so let's go and have a look how long this takes now this is going to take a couple of seconds so i've started looks like it's okay and you can see it's loading so essentially what's happening here um this is just a showcase of blocking if you like we can't perform any other tasks until that task is done and this is essentially why we want to potentially use asynchronous code so you can see it's taken nine seconds uh just over nine seconds to perform this action to collect a hundred items from the api these are the items that have been returned or the names of the pokemon cards and there we go so nine seconds let's now convert this synchronous approach into what we had originally the example we had originally so we'll just use async await and we'll see if we have any performance improvements over the traditional synchronous approach okay so i've gone ahead and just commented out the previous example you'll see that in the code that you can access through the link in the video description right so that's the previous so what i've done is i've just copied and pasted it and this is it right here so the only thing i need to change is to add the async function at the top here so that's the first change and then what i need to do now is just set up my aio http client session like we did previously in the first example and now just run my for loop again so i want to pass in the numbers 1 to 100 to collect all of those pokemon and now what i need to do is just uh create my url so i just prefer prepare that ready um and then i run the async with session get and then i run the url and then i return the results so i fire that off with a weight and then i return the data and then i can append my list here with the the names of the pokemon so i can just view them on the template and then everything else is exactly the same remember we've started the timer we're going to perform this action and then we're going to count just to make sure we've received all the data and then we're going to count the time that it's taken and then we're just passing that over here in our context over into the template right so the same process again so let's have a look to see how long this took now the first time utilizing a synchronous approach remember it took nine seconds so let's just give this a go now utilizing just a async await so this shouldn't take as long so that takes three seconds so you can see straight away we've over half that the amount of time it's taken to receive that data from the api and that would be quite consistent if we were to run this test a hundred times if you wanted to try that you would find that it's um going to be very similar the results that you're going to receive so now we're going to set up a third example we're going to bring in some of the tools from async io and that's going to further improve the performance time to collect all that data from our api and return it to our template so just to recap what's happening so far in kind of general terms in the first example we were using synchronous code and that was a process where we had to individually send a request one by one and we had to wait for the request to return in order for us to actually fire off another request for some more information and of course that takes a long time because we're always waiting before we can then send off another request so with async here in this example really we still had some block in here because we are waiting so we were still waiting we were sending a request and we were waiting but it did speed up we saw that it did speed up the process quite considerably so what we can do here in the third example is we can try and avoid this blocking com completely and we can now use some of the tools with async io to essentially just fire off all the requests at once if you like or concurrently in the last example we're going to be utilizing async io gather so this is going to run co routines concurrently now what we're going to do with the code is we're going to pre-prepare if you like all of the tasks that need to be completed and then we're going to fire that off with async io gather and without any blocking the tool it's going to essentially fire off all those requests to the api concurrently and return all the data and then that's going to be displayed in our template like before because this method isn't going to provide any blocking or waiting for us to send another request this is going to be working concurrently so that means essentially this should be a quicker way of gathering large amounts of data from an api and returning it over the previous examples which had an element of blocking an element or waiting before sending another request so we're going to start off with a similar setup i've just highlighted the code in the previous example here copied it across and just removed the middle section component here so this is what we end up with so copy and paste it remove the middle section here so now let's build upon this and see if we can now utilize the async io tools to speed up the performance of our requests so first of all let's uh bring in a new list called actions so what we're going to do here like i am suggesting is that we're going to develop all of the actions that we want to perform in our list and then we're just going to fire them all off and get the data from our api so here we're going to bring in like we've done previously um we're facing the aio http client session and we're going to run our full for loop as before um with a range of one to a hundred so we're going to collect the pokemon cards 100 and we're going to need the urls like before and this time we're going to have an action so action.append okay so what we're doing here is we're appending the list with some new data and we're going to set up here we're going to set up essentially the request that we want to fire off to our api so you can see here we use an async oh in short future and then we're going to need a new function here called get pokemon we're going to pass in session and url to actually get the data so this is going to be a function essentially which is going to perform the action of getting the data from the api and we're going to return the data from our api and now we can go ahead and fire off the asynco.gather so now essentially what we've done here is we've prepared everything that we want to collect from the api inside of our list here we're now going to use a cinco gather and now we can use this on a list so we're using star action so this is referring to our list here so we're going to essentially move through that list and get all the data now from our api that's going to now return in pokemon res results and from there we can use a simple for loop just to append our pokemon data list so we're just going to update our list with all the data like we've done previously with all the data that's returned so everything else else sorry everything else is exactly the same here we continue counting to make sure we've returned all the data we'll calculate how much time it took pass that back into our template here so no changes here and there we go right so let's now go ahead and fire this off so i'm just going to now refresh this page and there we go so there it didn't even take one second so instantly we can see the performance increase over the previous examples so there we have asynchronous django so hopefully in the future there'll be plenty more asynchronous django tutorials it will be interesting like i said to have a look at aio http with django on web sockets i think that would be an interesting tutorial to look at so hopefully in the future that will be the case so thank you very much for listening hopefully it was useful and i do hope to see you in another tutorial

Channel: Very Academy

Views: 4,828

Keywords:

Id: 28KFBqi2JrA

Length: 20min 32sec (1232 seconds)

Published: Tue Jun 22 2021