Async in practice: how to achieve concurrency in FastAPI (and what to avoid doing!)

hi this is yugini uh so when it comes to dealing with asynchronous Frameworks if you never dealt with them before it might be confusing and the documentation often doesn't help dealing with this confusion because there are way too many concurrency related terms a plus overall it's relatively new thing not something that's been a while since forever so and not everyone uses it uh so sometimes it can be tricky to navigate this space altogether and so one of the approaches that follows because of that is just writing async in front of every function and just hoping for the best the other thing some people do is just writing up synchronously like they used to and ignoring this part completely uh I was also asked in my previous video why I didn't use async functions for tutorial and that's a good question uh and there are actually a couple reasons for why I didn't want to do it uh first I didn't want to make the video more complicated but today I want to talk about this topic more in depth and show how it works in fast API with some practice I will make some basic benchmarks and I'll try to explain some theory behind it uh the most simple as I can so let's get to it uh here we have some endpoints and I will quickly show you what they got what functions they're doing and how it works overall so here in our code we have uh basic first API application uh that has those routes there is URL list is just list of uh websites for which it will try to connect during some Network process then uh this one will run function on a separate process we'll go to it later and there are several types of things that we can try to do asynchronous the first one will be emulating CPU calculations some blocking CPU band function and it will also it take a file and just emulate some things that it does with the file it will flip instead of doing something heavy on heavy computations and this one is written with async it doesn't use async slip because CPU band will always be blocking like that it won't be fair to use the Sync here the other one does exactly the same but is written with the regular function with regular style that I think next thing you have is sending some Network requests one would be sending a synchronous one through httpx it will just send a request to all those URLs that we've seen asynchronously and the other one will be doing the same thing with a sync keyword but with regular request style Library uh just uh synchronously normally and then finally we have a file reading it will be also a synchronous and asynchronous we will just get the file uh and try to upload it to our server server will try to read it and when it's finished it will just return done nothing else read file just uses regular style like you're always doing python a synchronous version will use Library called IO files that does this stuff asynchronously at least add claims to and now let's look at our functions so uh and the most important function that we have is greeting it will just send us hello async world and we will try to access this one this end point while uh trying while running other endpoints and we'll see if it will be blocked or if it will return at the same time so all our endpoints are combinations of reading file plus doing something else since we're trying to do asynchronous on the whole point is to make several things at once we'll try to read file asynchronously and just try to do other stuff and see what's going on first of all let's try that uh to the a thing thing that uh ah there's one more uh route that will try to read files synchronously but the function itself is written is written with async uh it's like something you might do when you don't know what to do you will write it hope for the best so let's see how it will work it will show us how long it took as well so let's select the file let's select some verse no let's select the Big File actually so it will take some time to execute and while it's executing we'll try to read and it returned at the same time so you might be thinking like well it seems that asynchronous keyword works you see yeah it's returned deployment done and all other stuff so it worked fine and didn't block anything but now let's see the regular read file synchronous that will use our synchronous function without a sync keyword as well and execute it and now let's try to grid and it also returns right away so I'll clear and show it also returns right away and here it also doing the same thing so then does it actually make any difference if we wrote uh asynchronous read file or synchronous one because the the this one just uses the regular one so what's the deal here so before we explore what actually is going on let's look at the other functions like this one so this file will this one will read the file and then use some calculations our CPU bound function that I showed you before but it will be written with the async keyword now uh read file and calculate synchronously yeah okay so let's take any small file actually here because it so it computates that stuff and now look we can't grid because it was blocked by the CPU even though you wrote asynchronous in front of it so it actually didn't work asynchronously we can try it again with anything else like and it still doesn't return so okay now let's try to do the synchronous version of the same thing that will do the very same thing but just with uh without async keyword in front of it we're still uh we can't await our read data file now so we'll just use the regular read file data and try it again so where CPU blocking and we get the the other one working at the same time without any problem uh and it all does and so not just the thing here uh I think keyword actually didn't let us didn't let our application to be asynchronous so let's see more examples of what can be uh achieved with the async keyword uh let's try uh requests first of all there will be a sink a sync keyword in front of regular requests yet that you use as normally and we'll use that and reading file asynchronously uh so yes this one so again we're choosing some file we're executing and we didn't get it uh at the same second like uh we did before and now look it took three and a half seconds and it also blocked our greeting so now let's try the ICC I think version of it the realizing version which is uh which uses httpx awaits mix different tasks with the same URLs and basically does the same thing but truly asynchronous use the same so previous function took like four seconds instead now this one took just one second and it also actually didn't block the main thread one second won't be enough to load both endpoints at the same time but we'll try your sleep so we'll sleep for extra five seconds to give us some time to check if we can still access the hello world endpoint so we're loading it and it returns at the same second so it worked actually much better it was faster and didn't block anything so now let's see why all of those things behave the way they did and what we can do to optimize the usage of asynchronous in first API so what we must note here is that there are types of of blocking one of them is CPU bound blocking is when your computer does some calculations and it's really busy processing uh something so it can do other things and the other thing is uh input output bound blocking input output bound blocking usually just awaits it basically Waits For Stuff to either get through Network it it also includes databases if you connect to your SQL database you're waiting for query to perform there or it can be Network you're sending some you're trying to connect to some API or website or it can be just regular file reading and uploading so with CPU bound blocking uh I think IO as you probably might have heard already won't help you that much but with io-bound blocking this is why it's called basically sync IO it might help you by by passing waiting time um so if a thing is IO bound blocking uh when it regularly what it does in its uh sequential programming layer of red fully consumed CPU uh as you can see here but if you are using uh some asynchronous things uh functions then uh it can just switch to the other fret while awaiting is happening or it can run several things at once but the important thing here is that it actually always happening in one process in one fret let's see uh what happens uh in fast API behind when you are trying to run endpoints with the synchronous keyword so we went to uh uh fast API and it's routing and you see here it says run and point function and it looks if what you are passing is correcting so basically if you wrote that it is a weight function and then it will be true and if it's a quarantine then it just calls it and the important part here is if it's not core routine then you it's run in fret pool uh what the Fret pool means well red pool is uh basically you have a process and inside that process there are separate threads that can do different type of stuff uh at the same time I will talk about it a bit more in depth it's like in a minute or two but uh the important part here is if you wrote a weight then the task won't be run in a thread pool it will be awaited so if you are trying to pass something that cannot be awaited or that is actually blocking and write a sync function then it will block the main event pool it will break the the main thread that's right in the event Loop and that is why exactly when you write a sync without actually using something that is true lacing deep down and you might actually be making things a lot slower than they were before so now let me show you something that I made to demonstrate that the best I can so let's imagine you have several tasks if you're doing it the regular way uh this will be a task and it will just be execution time uh it will just finish one then finish one start the other and like that until it finishes then there is that non-blocking that asynchronous way um it does this takes all the tasks then you use some function like a sink IO gather which we're using in some of our uh like here what we're sending request so we're making several tasks that will uh of sending something to URL then we gather the task away and await them then it will start it will just make it all five tasks in one big task basically and then they will be able to start at the same time so you won't be waiting each one of them to finish and that's why it takes uh way less time so it finishes and important thing here is it should be well truly synchronous it uses event Loop uh well what is the event Loop actually event Loop is just a fancy word for uh while true Loop which every cycle that runs it asks is it finished yet is it finished did it finish and so forth and when it's finished it sends the it tells the executor that the some task was finished if you uh it only creates the illusion of concurrency as I wrote here uh because actually the there is no like many there isn't many threats there isn't many processes so imagine like this block that we have here is one fret uh in one process and this one just runs event Loop and because of uh that it might bypass some weight so uh you will be basically just waiting for stuff at the same time instead of one after another so the advantages where it's pretty simple to use if you know what you're doing and it's lesser prone that frame which I'll talk about next uh minus is that you need to kind of know what you're doing because as I showed the writing async in front of it just doesn't cut it so you need to use asynchronous libraries and functions and should won't help you with the CPU bound tasks because they will still block doesn't matter how you write them then there is multi-freading so what is the motive writing well basically now on your imagine that this green block is a process it's a virtual or real like you have two cores on your CPU and four frets and imagine that this is one of those one of those four virtual course or two chords that you have uh it will connect this will be the memory of that CPU and you might run several threads on it that will all execute each own thing but they will all use the same memory that your CPU uses so they share a memory within a process one of them can be asynchronous like in our case so and what we saw a vest API does is that it go it asks uh for endpoints to if they are not as if the function is not the synchronous then they are run in a separate thread that's why you can actually have like 10 incoming synchronous request at the same time uh because just each one of them will go to different fret so it will be actually asynchronous even though you're just set the regular synchronous endpoint and because first API does it under the hood it does the asynchronous stuff using threads now while we don't use threads all the time for everything else well it makes it much more it makes programming more tricky and complicated because it can cause Deadlocks and race conditions so several threats might try to change the same values and then God knows what happens and it will be very hard to debug and understand what happened in the end and then there is python Global interpreter log then switching between threads also takes some uh time and some power and if in some cases it might make it even slightly uh some tasks slightly longer takes a little longer but overall it sure will be more efficient if you need to have to process several input bound stuff at the same time and it also doesn't help us with CPU bound tasks as I just like uh previous things that we looked at and the advantages as well it shares memory so if like usually your functions needs to use the same kind of data then every thread can access the same data and each process can spend many friends threats to run different tasks but and it's also something when you set up G unicorn web server for example you usually specify the on the server in production how many threads you need for it uh usually it recommends something like two times the amount of CPU cores plus one and spending me and since threats also take up memory uh spending too many threats uh might actually slow down your uh computer quite a bit so it's not recommended to create too many threads it's something that you need to have resources for and next we're going to just multiprocessing so which this is uh like those same like a multi-freading I told you that this is a process in which those are the threads that I show you here and those will just be separate cores let's say of your uh CPU which each has its own cache its own memory access and it is restricted by physical processor cores so you can't have more than you actually have CPUs so but the advantage here that it finally allows us to run multiple CPU bound tasks concurrently so at the same time you can calculate three times faster if you do multi-process uh and it doesn't share memory but it is which makes it well harder to use for tasks that are directly coupled so those are basically the types of concurrency that we have I tried to illustrate the best I could I hope this uh things that I drew make it easier to understand because when you were just reading those threads course multi this multivather think it just all turns into one big porridge in your brain and you don't remember which is one which one is which anymore in the end so and now let's get to what we should actually do uh at least what I understand we you must do when using uh concurrency and asynchronous programming in uh fast API for example so first of all file reading if you use it synchronously like regular it will go to fret so it won't block if you use it asynchronously with the i o files which is a sync library it also goes to Fred because that's what the i o files does underneath if you look if the source code it just does the same thing that test API is doing for you when you are writing the regular stuff it just does it implicitly so it actually doesn't matter at all which one you use and the weird part is that even if you write a thing here it also doesn't block as I showed you so they are here I didn't see much difference at all in terms of uh I think it will just goes to Fred either way so file reading is that way then I will go to the quest and networking well if you're using uh something that is a support synchronous request regular a library that most people use it doesn't support a sync keyword so if you're using regular one just use it regularly don't write a sink in front of it you'll just block the main thread and that's overall something that can be said about all the libraries if it's not I synchronous Library don't write this thing in front of it if you don't know what if it is just don't write a sink in front of it because otherwise you will make it slower because you will block the event threat of the test API because it wants add inst if you write it with a regular way it will send it to Fred so it will actually be a synchronous without writing a sync so that's where some confusion might stem from if you don't know what you are doing so and the best thing is sure to use a sync with and gather tasks like here so so first because if you just await each task one after another and don't use the async io gather like here it will still be like regular it will just do one after another only if you're Gathering tasks then like I should here it will do all the stuff concurrently which is what we want so overall use something that supports I think either https or IO HTTP or something like that that's cool so if you don't then use regular without a thing here so like that and then the databases what about databases in my previous video where I showed tutorial of how to do I didn't use any async databases it was regularly database so there is no point in writing a sync in front of every endpoint or function that uses database that doesn't support the sync as we know already and the thing is if you don't need to very if your application isn't something that needs to be accessed by like a lot of people like thousands of people uh a minute or second or an hour at least then the chances are it will just be easier to use regular databases and write it regular synchronous way because probably the regular workers will do just fine with it if you need a higher load application then use a synchronous databases for it you can't use the regular wrappers like sqlite or psycho PG instead you need to use the special ones and I wrote here some async Library so request is the io HTTP and HTTP hex ophages I think PG there is a library like that then if you want SQL Alchemy support but only the core part of the library so if you're using orm with your database then there is a library called just database just Google it you almost see it if you are using radius then there is IO radius for it so don't just write a sink in front of a synchronous regular libraries and wait for magic to happen because you heard that fast API is a synchronous you actually need to use the asynchronous capable libraries with it as well or just don't use a sync word at all cool and next thing we're going for is calculation so what you do about calculations I also um I actually wrote down here so first of all if you can't catch computations then you just should do that maybe if it's inefficient you may be just redoing the same thing over and over and then it doesn't make sense and caption can actually reduce the workload quite a bit then if you're using just some regular small computations it's a calculating something like for one or for a second then just use regular synchronous stuff you don't need it then if it's like something that is blocking for maybe couple seconds or so then it's best to use background tasks for it so let me show you what background tasks it's something that we are also import from Fast API and let's see the uh our endpoint which uses text so we import it as a dependency injection for our function and here we use background tasks add task and what we're trying to do so uh here we're trying to calculate data for file synchronous that one that takes six seconds and blocks of ram regularly and here we were also waiting for our uh I think read file data so let's try to do that and see if it blocks the Fret like it regularly did when we didn't use the syntax to refile and calculate synchronous oh again we're sending this it returned right away okay it returned right away that's the thing by the way uh that you want if you don't need return data then you can use it because uh but this uh calculation is actually still happening but it's happening in the background so you can't actually return something right away so if only if you uh write the result to let's say database or do some file or do something but don't need to send it back right away then use background task so if you as you see the cool thing is it just returned super fast right away now let's uh try to upload the bigger documents so it doesn't return right away and see if it's blocking now grid uh returned right away even when this is still loading so which and the other one went to the uh calculating uh so it doesn't Block Main thread and it also returns instantly so it's a good thing to do when you have a CPU bound test which doesn't need to return right away so that's pretty cool other thing that you can do is uh to just send it to uh explicitly to a separate process and to send your task the CPU Banton here I'm trying to do that so I uh made a function called run and process and it's uh yet the event Loop of rising IO and then it awaits it running in executor like that so uh let's see so I'm creating again uh tasks that I'm gathering later I'm reading file data and I'm trying to run this process with these arguments in the same thread and pass into tasks Now read file and calculate synchronously it's also pretty uh cool thing now look let's say I am important this thing so it doesn't block the main thread because you see it returned right away here which is good because the regular uh because regularly if you just write the CPU bound thing it blocks main thread so it didn't block it didn't return right away like background tasks did but it returned us the result after it was done so it says it took six seconds which was our CPU bound task how much it took and we got returned we got our async function and it just was uh doing it in a separate process so it's also cool thing that uh might be handy when you have some average size not very heavy computations that you still need to return them but don't want to block main thread with so yes and uh in the end what you can do uh if you have what math what you must do if you have a large actually some large CPU bound tasks that take maybe dozens of seconds sure you can't allow it to block your whole uh application uh because it just won't be able to do anything so for any heavy computations just also don't use background tasks because it's kind of not a good way so the industry standard is to just run it in a separate worker and use usually use salary and radius or rabbit and Q with it for that kind of stuff so yeah use that okay so here will be a quick recap I know the video was actually 40 minutes long and not everyone has that kind of time but I tried my best to not uh waste your time and explain why exactly some things are the way they are so that you just uh get some sense in how to how it works overall later so if you have time watch the whole video if you skipped uh previous right okay so let's sum this up if you have async functions use a sync library if you're using async functions gather tasks so they run concurrently if you have computations run it in separate process so use the worker for it or run them as a background if you don't need the return right away uh or at all rather if you need to read file data or something uh just you can just read it regularly it will send it to Fred don't write the sink in front of stuff that isn't actually I think uh what else ah one other thing I can just mention is actually when you read in file data here I'm just doing fire read but uh read it in chunks because if I'm sending let's say two gigabyte a file in the end it will take two gigabytes of my memory and M memory isn't endless so maybe someone will send a 14 gigabyte file and maybe you don't even have that much memory then it will just fail on the end so uh read it in chunks uh like small uh it's just makes more sense to read it like as one megabyte or eight megabyte chunks it might actually take slightly longer but the it's just much better to do it this way so so what else we got database if you don't and need a very high load stuff just to use regular database and without async just will be easier and it won't make much difference to you anyway so concurrency can be a little tricky as you see I need to also kind of tricky for me and I've been also struggling to understand all the concepts myself because every thing is so just confusing about it and the most resources don't help and also there are a lot of uh kind of just bad videos with those bad practices like using synchronous stuff with I think keywords something like that we see why it's a bad idea now so yeah I hope the this video was helpful for you and leave a comment see if you have any questions or if you have any input on the subject yourself yeah maybe you know some other things that I didn't mention uh or maybe you want me to talk about some other topic that I can explore in the future videos so thank you for watching

Channel: Evgeny Maksimov

Views: 12,134

Keywords:

Id: KL6CjNxkZDQ

Length: 40min 43sec (2443 seconds)

Published: Thu Oct 13 2022