An Example of Celery in a Flask App With Multiple Files

hey everyone in today's video I want to show you how to get celery working in a flask app that has multiple files so basically if you organize your flask app in the recommended way I want to show you how to use celery with that and the reason why I'm making this video is because I realized in previous videos my celery examples were too simple so in this video I want to make it a little more realistic so you can use it in your apps if you want to use celery for anything so before I get into the video I just want to say that I do have a coaching program so if you need one-on-one help with your project you can go to prettyprinted.com coaching or the link in the description below to learn more on how you can talk to me 101 and I can help you build your app or just answer any questions that you may have about building an app so what we're going to do in today's video is convert this app that I have which basically adds a user to the database and it does it in a long running process kind of way so it takes about 10 seconds to add the user to the database I'm going to convert this app to something I use as celery so as soon as you go to add a user to the form and submit commit it will immediately trigger salary and salary will run in the background and I'll have the options that cancel the task and everything so celery is going to handle actually putting the user in the database and then it's going to handle like the delay that I'll show you in a second so first let me show you the app that I have so this is the app uh it's pretty simple so here I'm just going to put my name add user just go there and you see it's kind of working it's kind of hard to see but it's it's working right now it's kind of loading it's it's waiting for something to happen and I put a sleep of 10 seconds in there that's why it's doing that and now 10 seconds has passed and the page has completed so now let me go over to my database let me open that up and just select star from user and we see that I have one user Anthony in the database so the way this works is I have this route has a form once the form is valid it calls this function called add user which I have in tasks stop pi and this ad user basically takes the form puts the name from that form into the user model instantiates that passes it's a database and saves it and then it sleeps for 10 seconds so the sleeping for 10 seconds is completely unnecessary but the purpose is just to show you how you can use celery for this so when I have celery working you'll see that this is being called in celery and your app won't have to wait for the 10 seconds to pass before it can continue working so let me start here I'll start by converting this function which is just a plain function to a function that can be used in celery so what I want to do is I want to import from celery I'll just put it at the top here the first and most important thing that I want to import is something called share task so from celery import shared task and the reason why I want this is because I won't be able to use the celery object to decorate my function to declare it as a salary task so in my other videos I always decorate the functions with something from the celery object and that way I register my tasks on celery but in this case I can't do that because of my project structure so I have this Dunder knit I have create app in here I'm going to instantiate the celery object but I have no way of getting that celery object to this test stop Pi so share task will allow me to declare my tasks in a way that's independent from the rest of my app and then when I do something a little bit later you'll see how celery kind of connects the two so I have shared tasks and I'll decorate this add user function so share task and what I want to do is I want to put bind equals true just so I have access to self inside of the task which I'll show you in just a moment and I also want to make this an affordable task so I can use another route to kind of cancel this task uh Midway through just in case I wanted that ability so I need to import something else it's from celery.contrib.abortable import a portable task and I'm just going to take that and I'm going to put it as base so if you're not concerned with aborting the task while it's running then you don't need this but for this video's purpose I'm going to have that just to demonstrate how that works so I have this task now this is now a celery task it's no longer a plain function so I have to change some things about it so the first thing I have this bind so I need to put self as the first argument because that's what bind does it passes self as the first argument without me having to do anything the second thing I need to do is change this form so right now this is a form from flask WTF or WT forms but this won't work because I'm going to be passing the form from my flask app to the salary task and you can think of them as completely separate in fact they can be running on two completely different servers so I need to be able to get a flask form over to this other server and to do that I need to pass it in a way that's something called Json serializable and that just means it's something that you can put in a Json object and represent pretty easily now form is an object checked so I can't pass a form inside of a Json object but what I can pass instead is the form data so I'll just change this to form data and then when I instantiate the user here instead of using form.name.data I'm going to use form data and then name right so I just need to make sure I pass the form data from the form which is pretty easy to do and then inside of the task here it's going to instantiate using that form data next what I want to do is I want to check to see if the task has been aborted that way I can just stop everything so what I can do in here inside of the for Loop because this is where you would want to check for something like this I can say if self dot is underscore aborted this method on self and I can just say tasks stopped right so what's going to happen is every iteration of this Loop which takes one second because of the sleep it's going to check to see if the task has been aborted somewhere if it has then it's going to end immediately and just return task stop to the celery command line okay so the task is done so let me go back to views because I need to update this to use the task so it's still coming from the task directory so now to change this I need to pass the form data instead of the form so I can just do form dot data here and that will pass the form data and then I also want to call this using celery so instead of just calling the function because that will work like before you have to call it dot delay on it and then the data that I pass is exactly the same finally what I want to do is I want to get the task ID so I'll return whatever delay returns into a variable call task and then what I'll do is I'll change this redirect in a moment to take the task ID and generate another page where I can potentially cancel the task but for now I'll just leave this like this so now with the view done in a task none now I want to actually bring in celery so what I can do is I can go over to my browser and I'm just going to open this link so this is from uh the flask documentation and I'm just going to take this code here so this is the code that they have for celery and my goal for this video was to take this code and not modify it at all so there are ways you can get this working by modifying this code but I thought it'd be easiest if I can get everything working without modifying the code that comes from the documentation so I'm just going to copy this and then I'm going to put it into a file called utils so of course you can name this whatever you want I'm just going to call it utils and I'll just paste it in there so I just pasted it in I'm not making any changes to it and I can just close it because that's all I want to do so now I'll go over to Dunder knit and I'll go ahead and import that so from Dot utils make celery and I want to update my configuration so I have Docker compose running and I have a redis instance so I'm going to use redis as both the broker and the result backend if you're familiar with these things in celery so what I'll do is I'll just put this under my secret key I'll say app config and then celery underscore config and that name comes from the utils so celery dot or celery underscore config that's what it's going to be looking for and this needs to be a dictionary so one key is going to be the broker URL so broker URL and in my case I'm using redis and Docker compose so redis colon slash slash redis and then I'm going to use the same thing as the result backend so result underscore back end and then redis colon slash slash redis okay so I have that now I want to use this make celery that I copied from the flask documentation so since this is like an extension I'll just put it right underneath my DB init app so what I'll do is I'll say celery equals make underscore celery and then I'll pass in the app object so it kind of works similar to db.nit app except that it's going to return its own thing here and then the important part that I have to do here is I need to call celery dot set default so what this is going to do is it's going to set this app so the app that I'm declaring this in as the default app and then any task that it discovers inside of the tasks.pi directory it will be added to this app so if I didn't have that then they will kind of be independent it would find this task but it wouldn't associate it with the app that I'm going to run so by calling celery.set default that's going to associate the two so all those tasks are set on the default app which is this one so so now what I need to do is I need to return celery from here so I'm going to return both app and celery as a tuple so just like that and because I'm returning this here I need to go into my run.pi and I need to grab celery from here so app comma celery equals create app because I'm returning two things now and the last thing I need to do here is I need to make sure that the application context is available because my salary task will do something with the database so you can't do this outside of the application context in flask if you're familiar with that issue so what I have to do here is I have to say app dot app context dot push and then what will happen here is when celery picks us up it's going to have the app context active so inside of the task it will be able to add stuff to the database if I didn't have this here when I would try to run a task it will complain that it doesn't have an active application context like for one it doesn't know where the database is because the the database configuration is defined on the app so by having the app context celery will now have access to all that information and it will be able to save stuff to the database and then the last thing I need to do to get this running is I need to go over to Docker compose so right now I only have redis and web and the web is basically the same thing as celery will be so let me just create a new service called celery and I can just copy everything and I'll make the modifications that are necessary so this needs to be indented just a little bit okay and what I want to do is I want to change let's see so flask app is still run flask debug doesn't really make sense in the context of celery the volume is still going to be uh Slash app in the container and the command is going to be a bit different so the command is going to start celery so it's going to be celery Dash a and then run so my run.pi DOT celery um and then I'll start one worker and then I want my logs to work so I'll put log level info so we can see everything working and then the ports aren't necessary but I do want to have depends on uh redis so I want redis to start before celery and it's complaining about something and I just need to de-indent some of these things here so this will work well and I think that's it so let me go ahead and try I will start up Docker compose again and here we see a task has been discovered so project.tasks dot add user and we'll see if everything is working so if you go to view stop Pi we have the delay and then like I said what I'll do is I'll come back and I'll uh at the task ID so we can cancel it but first let me make sure that this is working so I'll go back to my app and I'll reload the create user page and I'll create um salary user so add user so it immediately returns and now let's go back over to the terminal and we see it's counting up so it's hitting that Loop inside of the task where every second it just prints the second and then it says it's done it succeeded in 10 seconds okay so now let me add the ability to abort it so instead of redirecting to create user here I'm going to return the template so this will be render template and the name of the template here is cancel so cancel.html and I want to pass the task and let me make sure it's the task and not the task ID yeah so I want to pass the entire task to cancel so I'll just save that and now if I do this again let's say Docker user now at the user and the problem here is some redirects is missing so yeah I need the actual cancel function so let me go ahead and create that so I'm going to call this uh cancel and then it's going to take in the task ID and I'll name the function cancel as well and of course it still takes on a task ID so first I need to ask celery for the task so I can say add user by async results and then I can pass in the task ID so it will find the task that has this task ID that was initiated by add user and then I just want to do task.abort and I can return uh canceled just like that and now we can save it again and I think if I do this now uh last user add user here and now I see cancel task it's counting up I'll hit cancel it says canceled here and now I see here I have tasks stopped and just to show you the database again just to show you that I have those users we have celery user Docker user and last user so yeah that's all I wanted to show you in this video not much of a tutorial just more of you showing the process of how to get this working the key ideas here of course are the share task the set defaults where was it the set default here and then of course the pushing of the app context in your file run.pi so I hope this helped you if you were wondering how to get things running in a typical flask project setup and like I said if you have any questions about this you can always ask me for 101 help if you have like a small question you can leave it in the comments down below and I'll try to get to it so that's it for this video If you like this video please give me a thumbs up and if you have subscribed to my channel already please subscribe so thank you for watching and I will talk to you next time

Channel: Pretty Printed

Views: 12,296

Keywords: flask, celery, multiple files, create app, application factory

Id: 2j3em0QQaMg

Length: 15min 40sec (940 seconds)

Published: Fri Jan 27 2023