FastAPI Celery, Flower and Docker

hello my name is xander and welcome back to fast  api in this tutorial we look at implementing   celery flower and fast api in docker containers  so by the end of this tutorial you would have   integrated celery into an existing fast api app so  we're just going to take a previous implementation   previous tutorial fast api tutorial we're  going to take that code and just implement   celery so we'll run celery redis fast api within  docker containers so it goes through building   that up in this tutorial and then we go ahead and  implement flour to monitor celery jobs and workers so i am making the assumption here that you  understand what celery is might be used for   here is a visual kind of representation of what  we're going to be developing and for those who   don't have a general understanding what celery is  for then i'll just quickly give you an overview   by all means skip ahead if you're already familiar  with celery and you just want to go through the   process of implementing so let's imagine you  have an api um service that is quite data heavy   has quite data-heavy tasks that might take a  few seconds or even a few minutes to complete   the user on the front end makes a request that  was sent to the server now maybe traditionally the   server would then perform that action that might  take a couple of seconds might take a few minutes   and of course while it's doing that is taking up  the resource of the web server and if you imagine   if you scale that up to a thousand hundreds of  thousands that's a lot of resource from one server   that has to deal with all those requests and of  course the more requests that the web server is   dealing with potentially the slower that service  becomes when others want to utilize this service   so the whole point here with celery is which  allows us to kind of offload some of this work   from the web server in this case in this scenario  over to what might be known as salary workers so   this might be a separate server or service  that we can deploy in order to perform these   intensive tasks processing tasks that might need  to be completed and also when we're doing that   this will work asynchronously so that does mean  that users can make a request to the web server   and that the request then isn't blocked until  the task is complete the user can then continue   potentially and perform other tasks whilst it  waits for the the job to complete so to summarize   this workflow here the user is going to make a  request that's going to be picked up by the web   server this is all going to be on one container  here and then that's going to be passed that job   if you like is going to be passed across to our  message broker here and then our salary worker   which is going to be in another container so  redis we're going to put in a separate container   and then celery we're going to place in a  separate container a celery worker sorry   is going to pick up the message or the job from  the redis message broker it's going to perform   the action perform the task and then it's going to  be locked so then we're going to implement flower   within this container the web server container  here in this case and that's going to basically   monitor salary jobs and workers so what  we're going to end up here in this tutorial   we're going to end up with three containers  one for the web server fast api and flower   one container for a redis message broker and we'll  then also build a container for the celery worker   so use the link in the video description download  the code i'm just going to use the code from the   previous tutorial where we implemented fast api  with user token authentication so i'll just go   ahead and download that and i'll open that up in  visual studio code so once you've downloaded the   base code go ahead and run a virtual environment  so we're just going to hook into that environment   start that up okay so what we have here then  if we look in the docker compose file this   code from the previous tutorial if you haven't  seen that then maybe go ahead and do that if   you're not familiar with fast api in general  there is around about three or four videos kind   of leading up to this tutorial which you can also  look through to familiarize yourself with fast api   so here you can see that we're going  to be utilizing docker compose so you   are going to need docker installed docker  desktop we're going to be utilizing here   so please go ahead and do that if you want to  follow the tutorial we're going to add to this   docker compose file we're going to need to  add redis and we're going to need to make   our salary worker so those are two things  we're definitely going to need to do here   so we're just going to make a change here  in line one we're just going to use a slim   version of linux if you like so in the previous  tutorials we've been using just the default image   which was fairly large around about a gig so this  is going to slim things down a little bit here   uh so there's nothing else we need to  change that so i'm just going to save that   so looking back here in our diagram we're going  to need a container for redis so we're going to   put redis in a container and then our celery  workers so let's just go ahead first and build our   celery container sorry our redis container okay so  in docker compose let's go back to a dot compose   uh let's go ahead and add a new service  just down here so we want to add a redis   service so let's just call this uh redis and  then i'm going to give this a container name   of celery work now because it's ready sorry  i don't know what i keep saying sorry and   then we're going to need an image so if you head  over to the docker hub you'll see the redis image   so here we're going to be utilizing redis i  think the latest version is six uh 6.2 alpine   okay so that's pretty much it for sending  up redis in this scenario at least   uh so now when we build our  containers we'll find we have redis   so because we are using we swapped over to  a slim version here to a slim build here   um it does mean we will have compatibility issues  with the database that we're using the postgres   database so what we need to do is go into our  requirements here um well we won't be able to run our normal adapter postgres adapter we're going to  need to swap this over so simply put here because   we're using a slim version of linux for example it  doesn't have all the tools that we need to run our   normal postgres adapter here so what we're going  to need to do is just remove that from our list and then we're just going to  swap that over for the binary   version and there we go so as a  recording 2.9.1 is the latest version okay so if you are seeing this message error  message here when you go ahead and build   that'll be why so we can go ahead  and build so docker compose build   so that might take a few seconds because we need  to download the image uh and get it prepared etc   so you don't have to use the build command here  we could just use stock and pose up for example   and get everything ready so i've just done that  just to ensure that the changes i've made here my   docker file um it will build okay so once that's  done i can if i go back to for example here the   image you can see i've made the app image so again  if you're not too familiar what's going on here   then maybe go back to the previous tutorials to  give you a better idea of how we kind of built   up to this stage but we built our image and this  is essentially our api image with fast api and   our web server so now we can go ahead now and  add all the other services that we have ready   to go so that would be now everything here in the  dock and post so we're going to build our postgres   image um we're going to have our pg image  pg admin image to manage the database   we've got that the app and then the redis server  or the redis um container will also be made   so let's go for up so that's going to then  pull down all the images for these different   services here and get them all ready that'll  take a few seconds so i'll just pause here so while it's downloading that you  can see here that from the previous   tutorial this was around about a gig  the size of this image we have for   app and you can see now we're using the  slim build you can see that that's been reduced to over half the size so you may have  already noticed this that i made a mistake   spelling alpine apologies so if you've followed  that and you've made the same mistake i just need   to go into terminal again do the docker compose  up and that will then finish off the build here   so what you have at the end of the day here once  you've done doc compose up is we're going to have   four images these are the services that are  going to be turned into containers so redis   this is going to be our pg admin our admin for  our database postgres database and then we've   got our app here as well which which is our fast  api implementation our api so because we've used   docker compose up the container should now  be all running so you can see that our redis   service is now running as well as the other  containers so let's go ahead now and start to   implement celery so i will start by just  pressing ctrl c to close all the containers   now that will just exit all the containers  here i'm also just going to remove uh this   image here because we're going to need to make  some changes so i'm going to remove the app image   these can also stay the same that's not a problem  there's nothing else we need to configure there so   once we're back into the app here we're just going  to pip install celery that seems like a good idea   and we're going to be utilizing celery so let  that load up so once that's installed go ahead   and install redis so pip install redis here i'm  using redis 3.5.3 so you will find all this in   the requirements text of the repository for this  tutorial if you're not too sure what versions   we're using now when we're using celery and redis  here and working with these tools it's important   because things can change and break depending  on what version you're using so if you're not   too sure head over to the requirements text and  install the specific version we're using in this   tutorial so that might not be a problem if you're  watching this tutorial a couple weeks or months   after this was made but if you're looking  at this tutorial maybe a year after   um this tutorial was made then yeah just take note  of the versions that we're using so just before we   configure or start to make configurations just  a change here i've made to the diagram workflow   diagram here we're going to be utilizing  redis here to essentially save the results   of the tasks that we're going to complete so  whereas we had before i had here in this container   the results or storing results here we're going to  restore results in this tutorial utilizing redis   so next up i'm going to set some environment  variables here for our redis service so   let's go ahead and do exactly that so we're  going to define here our salary broker url   so that's going to be redis that's the name of the  service that we've set up here in docking pose and   that's going to be running on port 6379 um so here  we're just defining the salary broker url which is   obviously redis in this case um so that will point  celery over to the broker service and we're also   going to be using it as a result back end if you  remember so let's set up that environment variable   so this is going to be celery result back end  um redis same thing again right so that's the   two variables we're going to set for our  broker and identifying our result backend   so let's go ahead and build a new file  i'm just going to call this celery   celery let's just call it celery okay right  so in here i'm going to import os and then   import time it's going to use these tools  and then from celery let's import salary yep and like we did before if you've seen the  other tutorials we've been utilizing um dot emv   package in order to kind of load up our  environment variables uh so that would just allow   us to access our environment variables when we're  working here in in development mode right so um so   from emv we import uh load so this is basically  going to allow us to load the environment   variables and then for us to access them because  we've set up to have we just previously set up two   environment variables so now we can go ahead and  load them so that was a emv file now this is um   in the same directory location so it's just  dot and then the file emv so that will load   up those and then we can go ahead and use them  shortly so let's go ahead now and set up celery um so we want to define our um broker  url so let's just uh configure that   so you can see here i'm using os um environment  environ get so that will allow us allow me to   access the environment variables which have been  loaded and i want to get the salary broker okay   and i can do the same thing again um for the the  back end right so that essentially is now set up   celery so i can now go ahead and just add a task  so in this tutorial it's i'm just going to create   a very simple task which is simply essentially  going to kind of add some numbers together um and   return we're going to use um time to create a  um to kind of showcase the fact that it might   take a couple of seconds so we're going  to utilize it to simulate the task taking   a number of seconds for example so let's go ahead  and use the celery declarator celery dot task so name equals and then we'll  just call that create task uh and then we'll create a new function  called create task and we're taking some   parameters a b and c for example um and then what  we'll do is we'll just create a simple delay so   we define the delay that we want to create so  this is just gonna sorry this is going to delay   the task as if the task was  going to take a couple of seconds   etc so we're just simulating that  and then we can return b plus c so i do apologize i'm going to assume that  you've got a general idea of salary so here's   a task celery task it's going to we're going to  input a b and c so a b in the time we want it   to be delayed the task as if we were creating an  asynchronous task that would take a few seconds   and then we're just going to input b and c just  to return a result of a simple calculation so   that's the task that we're going to build so  ideally or essentially what's going to happen now   is this task is going to be sent across to  our broker uh redis and then our salary work   is going to pick up that task it's going to  perform that task and then return the results   back to our redis server and then we're going  to pick up those results using flower software   package to see what's happened  and the result that's returned   so let's go back into our docker compose file  here we're going to create a new service called   celery worker and we're going to give it  a container name and we're going to build   so here essentially what we're doing is um we're  just going to use this implementation here which   is going to have a salary installed etc to kind  of utilize these files as our celery worker   so we're going to run this command  here to kind of start our worker   and then just make sure something with celery  if you're looking at different tutorials   um once you move up to celery i think 0.5 or  version five sorry um you'll find there was   a few changes or big changes that were made  so if you are using different tutorials to   help you understand and to learn celery just  be careful of the commands utilize and just   make sure you know what version you're working  with and corresponding those commands to the   version of celery that you're working with okay  so here we've got our volume here so that's our   volume we're just gonna utilize um the code here  to implement this and we got the environment here   um so we're going to define the salary broker  and the result back end so that's going to be   taken from the environment variables and then  this is going to depend on the app and redis   so we don't really want to run this until we know  that the app is working and redis is also started okay so now we have a salary task here on our web  server we have redis prepared our container we now   have a salary worker container um we haven't  started that up yet but that is now prepared   so let's go ahead now and implement flower so  that we can um see and visualize um our tasks   or should i say monitor our tasks right so  let's go ahead now and pip install flower so once we've installed let's go back over to our  docking compose file let's create a new service so   flower container name is going to be called flower  and we're going to build uh command celery um   dot celery flower port 555 so that's going to  start everything up and then we go ahead and   sort out our port so we can access this service  this is a web service so we're going to access   that on port 5556 mapped across to the container  555 and then we're going to set up our environment   broker and results and it's going to attempt  depend on the app redis and celery worker   so here if you're not familiar with depends on  essentially here we're just going to create a   a container start order if you like um  so obviously this depends on these other   applications work and otherwise there's no point  starting this service okay so now we've done that   um let's go ahead and just double check  everything so we've got flower installed   or prepared um i didn't notice here i've called  i've set the command to salary worker and this is   uh essentially pointing here to our salary file  so i just need to rename that to celery worker   essentially our salary worker is going to run this  task now you've seen here in the implementation   in our docker compose file our celery worker  is basically just copying um everything from   this directory here and essentially the  worker is going to perform um the action   here in our what is now called salary worker  underscore worker is going to run this task so   we're just using these files here as well as the  requirements to kind of build that um container   salary installed it's got the worker task that we  need to perform so we're going to pass tasks over   to we're going to pass the parameters obviously  over to this worker and it's going to perform   this task so the attributes are going to pass over  and it's going to perform that task right so let's   just make sure that's all matched up so we've got  celery worker here to command and we're going to   start our worker and with flour so that is just  matched up here now with celery worker right   so if you now uh just make sure here what  i've done here i've got postgres redis   pg admin here you can see i don't have any of  the other images there's no containers there   so that's all ready to go so now i'm  just going to go ahead and compose up   so that might take a couple seconds just  to rebuild our our worker and our app   so you can see what's happening here  in the commands it all looks good   and you can see that celery is now running it's  picked up the task which was called create task   um so that's our task name in our salary  worker file here called create task so you   can see that celery is it knows about our task  it's ready to potentially perform that task   looks like flower's working okay so we can now  go into the browser so i've headed over to the   browser i've typed in the localhost 5556 that  was the port number we've identified in the in our docker compose file  here so we've set the flower   port here 556 mapped over to 555 in the container  so yeah it looks like flowers ready to um monitor our tasks that are completed so here we've  active processed and failed and succeeded so you   can see that uh what's essentially is going to  happen now is we need to perform a task uh task is   completed that's sent back to our redis container  of course and then we can pick up from flower here   we can pick up the results of that task so at the  minute there's actually no way of us to actually   run the task so what we need to do  now is think about this so let's just   go back into our example here so we need to  be able to now make a request to our server   to perform a task and then that task  needs to then be pushed over to our redis   message broker to in order for our salary worker  to pick up and perform the task right so we kind   of need um if you like an endpoint almost  here so let's go into our main file here   so what we're going to need to do here then is  we're going to need to be able to post some data   to our api so that we can then run our tasks so  let's go ahead and do exactly that so we have app   fast api implementation here so we're going to use  um at app.post so we're going to post some data   so we're going to set the endpoint here to example  one so that's where we're going to send the data   so let's go ahead and just  uh create a new function here   run task data equals body so we're  going to capture the data that we send   in data so amount equals uh which is going to  be an integer data amount so here what we're   doing is we're going to grab the data um that we  post across and we're going to grab the amount   okay from that data the quantity and that we've  set and then we're going to store that in amount   and then we're going to go for x and y so the  amount here is just referring to the amount of   delay that we want to create that's going to be  our a if we go back into our our worker you can   see here we're taking in three parameters here  a is going to be how long we delay the task to   simulate how long a task might take and then b  and c here we're just going to calculate right so   let's go back to main here so x and y is going to  be our um our numbers that we're going to simply   calculate right so task equals create task inside  of create uh task so task also create task.delay   amount x and y okay so that's our task and  then we're going to return jsonresponse right   so what's great tasks so what we're going to  need to do now is uh just bring in a few tools   so from celery worker so let's put it here from  celery worker now we're going to need to import our create task yep okay so that was the task  name in our salary worker it's called create   task so we're going to import that in um where's  i mean there we go so that's it so now we're just   basically utilizing that um delay amount x and  y so we're going to pass those in so those are   obviously going to be captured if we go back to  our workers we're going to be captures a b and c   i'm going to define the sleep time  and then we're going to return b and c so in addition let's also we are using body  here so we need to take that from fast api   we need to import that in so let's import that in  um you can see that we've received an error here   it's not been defined so you can see once that's  been done the app has now started up again   so now we have to fire a task off to our api so  here we can use curl um now i'm using windows here   so i do have get um bash as well available so i  can utilize this to run curl commands so here i'm   going to send a request a post uh to localhost  8000 that's where our server is and then we set   out a new endpoint called slash ex1 so content  type application json i want to send some data and   what i want to send here is i want to capture send  the amount which is the time delay and the data   x which is going to be 1 and y is going  to be 2. so when that gets processed by   our task if we go back into the work here  you can see that a is going to be used for   the time sleep so that's going to create  a delay and then b and c is just going to   record account called create a calculation  so one plus two is going to return three so give that a go and it says  here internal server error   okay so this is good so we have an error which is  always good to try and resolve this one's really   simple says here name here a name json response  is not defined so you can see here i've got a bash   um here a git bash shell open here um where i'm  running my curl commands so i'll go back to the   previous if you're not familiar with the terminal  you can see i've got a an error response here json   response so if i go into my main here all this  really means is that i've not actually imported   from fast api uh jason response so here  i'm trying to return a json response   um and of course i can't do that it looks like  everything else is working so my app is now   started up again so i'm just going to run this  command again press up just going to send the   same data across again and then this time you can  see i get a response task 3. so you can see that   i could change it to count calculation  or result i'll probably make more sense   so if i run it again it should now say  result is free oh probably saving time so your result is three all right so i  can change this up obviously so we can   confirm this is working so it changes the four the  result should be five right so four plus one is   five so it looks like everything is working okay  now if i go back into flower you can see that i've   processed six tasks um and six were successful  so we can clearly see that flower is picking up   um from our back end there the tasks are happening  and we can see this in real time so for example if   i had to change the amount here to 10 seconds  so that's going to be essentially a 10 second   wait if i go into flower here you can see that  we have an active task it's being processed well   obviously we're simulating this this task  might take an x amount of time to complete   so once it is completed you can see it goes over  to succeeded and the task has been been completed   okay so that should give us a starting point  to start working with salary and fast api   future tutorials we can now go ahead and explore  um how to implement different tasks or set up   different tasks of salary this should act as a  good starting point just to get us over the line   starting to use salary with fast api  so i hope you enjoyed the tutorial   thank you very much for listening and  hopefully see you in the next tutorial

Channel: Very Academy

Views: 33,976

Keywords: fastapi, python fastapi, fastapi tutorial, fastapi example, fastapi docker, fastapi redis, redis fastapi, fastapi flower, celery, fastapi celery

Id: mcX_4EvYka4

Length: 29min 32sec (1772 seconds)

Published: Fri Jul 16 2021