Background job processing using Python, FastAPI and Redis Queue | Python | Message Queue

hello people welcome back to my channel and in this video we'll be creating a very awesome project of background job processing using python redis q and fast API if you're a software engineer or you are aspiring to become a software engineer then in that case you will be creating Services which depend on queues and those are basically used for processing your background job so let's say you want to do some image processing you want to do some jobs which are scheduled so for those cases we need a background job processing system and in this video we will be creating that let's understand what we are going to create so there will be a queue and there will be a producer and there will be multiple consumers attached to it so the producer will publish message to this queue and then this queue will basically have some consumers which will consume the message and it will process the business logic so in the current case what we are going to do we will be using redis as a queue and then we'll be creating a fast API API server so fast API is a framework in Python which is used for creating rest apis and we will be using some of the apis on This Server through which we can post job get job status and other things and there will be a consumer or you can say worker also which will basically listen to this redis queue and then it will process the job that we have written so the prerequisitive that is needed is you need to have a IDE to code in Python since this project is in Python so we need something where you can code in Python and then redis must be installed you can follow my article on this install reduce using Docker compose where I have given step by step process to basically create a ready server and also a graphical user interface or you can follow this video where I am installing redis using Docker and Docker compose so I'll give the links of these particular articles and videos in the description below the prerequisitives are only these things next step is environment setup so we need to create a project in your IDE for python development and I'll be using pycharm so the pi charm should have a support of python3.x because we'll be using some libraries which requires python3.x interpreter let's set up our project in pycharm so I'll go to pycharm click on this new project here I'll select pure Python and then the location where I want to create my project so I am using WSL of Windows and here what I'm going to do inside my home I'll be creating a new folder and that will be called as sandbox inside sandbox I'll be creating a new folder that will be python background job processing click on OK this is going to create this folder and I'll hit OK the next thing is to configure my interpreter so I'll click add interpreter here and here I'll select on WSL so these are the steps that I am performing you need to follow the steps that for the ID that you are using I'll click on location here and here I need to provide the path of my virtual environment so here inside the sandbox I'll select this folder which we created just now and then here I'll be saying the new folder name is VNV I'll click OK create and this will create a virtual environment and our pycharm with the project is present here so you can see here there is a VNV folder so I'll click on the terminal I can hit here Source VNV bin activate and this will activate the virtual environment so the next thing that we need to do is basically we need to set up the fast API server so the fast API server is created using this fast API library of python and then we need uvcon library to run the fast API server the third step is the installation of Pi dentic Library this is needed by fast API to accept data in your rest request and finally we will be creating a fast API server instance so let's get on the terminal and start coding so here I'll create a new main.pi file so the file name will be main dot pi and here we'll be doing the stuff for creating the instance but before that we need to install fast API so I'll click on pip install fast API and then I need to install uvcon because this is needed to run your fast API server third thing that I need to do is pedantic so I'll install pip install pedantic and finally we'll be creating the rest API instance from Fast API import fast API and the app instance will be equal to fast API so this is the app instance that we have created let's go to the slide and see what is the next step the next step is to create a ping API on the server it will be a get route with this localhost port and ping is the route which will return as a response of success to message equal to pong why do we create a ping API because ping API helps you to check that whether the server is responding correctly or not so it can behave as a health check for your server so let's go on to the code and here we can say Define index and this index will be the folder where we'll say we are returning the response as success equal to true because this is the contract that we have defined and then message will be equal to punk but this is just a simple method if you want to create a route out of it so we need to use the app decorator and then get and then the path so the path is get so this is the first route that we have created now let's run the server using this UV code main app reload so I'll be using my terminal here because in the terminal it's easy to see the logs so I'll go to CD sandbox and here we'll go inside python background job processing I'll activate the virtual environment and then using the UV corn command that is uvcon main colon app main is the file name app is the app instance and then reload is basically used because we want to reload the server whenever we do some file changes next thing is to call this API that we have created on our server that is the sket API so I'll open my Postman and here you see that a get API call is there with localhost 808000 is the port on which our server is running I'll click Send and you can see here I got a response of success to and message equal to pound so the first step of creating a get API is done here let's go to the next step The Next Step says that we need to create this redis queue because our rest server will have an API which will basically push a job to this redis queue and there will be consumer which will be consuming from it so for this redis queue we'll be installing RQ library of python which helps us to create this task queue and then send the job and then do the job processing so we'll be creating a queue with the name task queue in this radius so let's go to the python here and we can install pip install RQ this will basically install the rediscule library and finally we need to create the redis queue instance so from RQ import Q and then from redis import redis create the redis connection and that will be equal to redis and these are the different parameters that you can provide you can see here host Port another thing so we'll just say host equal to localhost will have a ready server running on our local host and then the port will be equal to 6379 which is the default port and finally we'll be creating the task queue and the task queue will be q and the name of the task you will be task queue the redis connection will be equal to redis connection which we have created here so what we did we basically imported the redis class we created a connection out of it and then finally we are creating a queue with the name task queue on this radius so let's go on to the next step the next step is what it is actually the job that will be created so our job is a number printing machine so whenever someone will call the fast API server a job will be published to this redis queue so we are saying that our job will be a number printing function which will basically print number so we'll provide the lowest and the highest values and between those two inclusive of these two all the numbers will be printed by our job in pycham let's create a new fold a file that will be equal to file name equal to job dot pi and here inside the job.pi we'll just create print number and this is going to accept two parameter that will that will be equal to lowest and highest and here what we are going to do we are saying x equal to lowest because we are going to print the number so we want to do some looping so we'll say while X is less than equal to High s means X is smaller than or equal to High s till then we'll be printing the value so we'll say print and then slash n as a new line and then we'll say this sorry format and then X and finally when it is printed we need to increment it also so we'll say x equal to X Plus 1 and in each iteration the value of the X will be increased and finally it will come out when we start the uh this job so we need to print something like print equal to equal to job processing started and then finally once the job is done we'll say print and then we can just say end so this is the end text that will be printed once the job is done so we have created this job dot pi and here we have created this method which is the actual job so you can Define the job based on your scenarios it can be a image processing system it can be a job which can call to a server scrape the data and then put it put the things inside the database now once this job is created so what we need to do we need to create a post API which will basically accept the body parameter as low lowest and highest which will be basically numbers and based on that print number job will be executed and it will print all the values inclusive of lowest and highest so let's create this on our python application so what we are going to do we are going to say app dot post and inside post we are saying that the route will be job and we will create the method name as defined post job and it will accept a parameter so as we have said that we will be using pedantic to create the data that will be accepted in the API route so for that what we need to do we need to say from pedantic import base model and we'll be defining a model which will be basically the request data that will be receiving so we can say class job and then in job data and the job data will be inheriting from the base model and this will have two values one will be equal to lowest which will be end and then another will be highest which will also be end so this is the job data that will be accepted in the request body and here we can just say that the key or the variable name will be job and it will be of type job data so once this is accepted we are going to say that lowest value that we receive will be equal to job dot lowest because this is the data that will be sent in the request and from the data will be getting this lowest value and the highest value will be equal to job dot highest so once we get this data what we are going to do we are going to enqueue the job which we have written here basically this job with this lowest and highest number so for that what we need to do we need to import the print and number job so from job import we are going to import the print number so here once we have imported the print number what we are going to say that task queue dot enqueue because we are going to send some data here uh to the queue so if you see here so what we are saying that this queue has a task queue and inside the task queue we will be sending the data so task queue dot NQ and this will be equal to print number and the parameters that will be passed will be equal to lowest comma highest so this is the API that we have Exposed on our fast API server which will be enqueuing the job so let's go to the terminal here and you see that the server has already restarted we need to start the worker so ready skew Library itself gives the command that is req worker and then the Q name in the another pane what we are going to do we are going to do LS and here inside the sandbox we have this background job processing here we'll activate the virtual environment and then we are going to do RQ worker and then the task you since the queue name is Task queue so it got connected to the redis the radius is already running in the background it got connected to this and now let's go to the postman from there we'll be sending this uh API request that is localhost 8000 job job is the resource and we are sending post data inside the body we are saying the lowest value and the highest value let's see what happens so once we send it we are getting a null value because we are not returning anything from the API let's fix it first so I'll go to this and from here whenever we enqueue the task we get the job instance so we can say job equal to this or we can say job instance equal to this and then finally return the response as success equal to true and then job ID equal to job instance dot ID because whenever this RQ publishes the job to the queue then it gives a unique ID to that job let's go to the terminal again you see that the application has already restarted and you can see here we are getting five six seven eight as printed Also let's change the parameter so we'll say we want to print from 10 to 20 and let's hit send so here you can see this time we got success equal to true and then the job ID this is a unique ID which can be used to access the status of the job if it is long running let's go to the terminal and here you can see job processing started 10 11 12 13 14 15 16 up to 20. so what happened basically is the rest API was invoked on the fast API server which basically published the job to the queue and then finally the job was processed using the worker this is what we created in this particular video this was the sample project where we created fast API server then redis queue was used for publishing the job and then the consumer was also created it consumed the job and printed all the number so please comment on this video like what things you liked about this particular project if you want to learn some new project please post in the comments so that I can create those projects for you and it will be very helpful for you you can post in the comments also like what type of new videos do you want and you can give suggestions for me also to improve so thank you for watching this video please subscribe to my channel and please share these videos with other people thank you

Channel: CodeWithRajRanjan

Views: 5,429

Keywords: redis queue, python, fastapi, background job processing, message queue, aysnchronous message queue, job processing

Id: 12eNGDjXY7s

Length: 16min 7sec (967 seconds)

Published: Fri May 12 2023