How to Make 2500 HTTP Requests in 2 Seconds with Async & Await

so when you run your code it will wait for each line to be executed or each block of code to execute before the next one can happen now if you put something in there that requires a bit of waiting time like you're waiting on a server or a response from something external your program's just gonna be sat there doing nothing now there's a couple of ways around this we can use threading and we can use async i'm going to try and show you a few examples using a request to a server so i can show you how it all goes a bit quicker so on the screen i have a synchronous version this basically just goes out to this url sends a number uh one to 201 to two to two and a half thousand and then gives us a response back now as you can see uh when i ran this it took 5.4 seconds so this is what i'm actually using as my demo it is a server on the local host on my machine here on my network so we're actually not getting an awful lot of delay in the actual response but that's just going to further compound so you can really see the difference in time taken and then you could extrapolate that out when you actually have to wait for a response back ignore this one we're just hitting this endpoint which is just returning an id a number so when i run this code here it's going to go through each line line by line when it gets the response back you can see it's printing it out to the screen here and it took 5.85 seconds slightly slower than the time i ran it just a minute ago so this is all well and good but what happens if we have thousands and thousands of urls that we need to check on this is going to take forever so this the met the first option is to use uh threading so i'm going to be using here in this example we're using the concurrent futures we're using the thread pool executor which is basically going to allow us to use extra threads and it's going to speed it up ever so slightly so generally speaking threads and multi-threading like this it's better for if you're trying to work in parallel so if your code is constantly doing something you're not waiting for anything external then this is probably going to be the best way for you but if you're dealing with any network requests then you bob you are going to want to use async and i'll show you that in just a second so let me just run this one quick you can see it looks slightly different maybe a bit quicker and this is going to come through 3.84 seconds so slightly faster than that one there this is fairly easy to use and fairly simple to set up all you need is a function that does your thing for you and a list in this case because i'm using the map there are different ways that you can use this but this is how you would use it if you were making network requests so the third option is async now this is slightly more complicated because there is more to it we need to have an extra function here so we can see that this is our main function that actually makes the request then we have one here that creates all the tasks we are using async io here and it will await for the response for us so this is basically like sending out all of your requests in one go sort of and it will manage and wait for each of the responses to come back so it will basically remove any of that time spent waiting in your code so i'm going to run this and we'll see that it will take literally about a second 1.37 seconds so it took 1.34 last time we got all of the data back within a list so when would you want to use any of these versions well if you're trying to request data from the same server over and over again you're going to find that you're going to get rate limited so you aren't generally going to be able to use this async version but you can work it through because if for example the rate limit on your server that you're trying to get to is 300 calls per minute and you only need to make 200 calls you can asynchronously send 200 and get all the responses back in a flash whereas in if you're doing it synchronously you'd have to send 200 actuals two or 300 single requests which could take a couple of minutes the other alternative is if you have lots of urls that you need to visit for lots of different sites lots of different servers you can then use your async code to request them all in one go and get the responses back and then handle it that way so there'll be a link to my github down below with these examples in and if you're interested in how you might use these in a different kind of way i think you're going to enjoy this video right here

Channel: John Watson Rooney

Views: 46,702

Keywords: async, python, async await, asynchronous, aiohttp, python requests, python web scraping, john watson rooney, http requests, python http request

Id: Ii7x4mpIhIs

Length: 4min 26sec (266 seconds)

Published: Sun Feb 27 2022