Asynchronous Tasks in Python - Getting Started With Celery [2020]

hey everyone in today's video I'll introduce you to celery and Python so celery is a distributor task queue which means it allows you to run tasks from your Python app or your script or whatever you have which are basically just functions it allows you to run these functions or tasks asynchronously so instead of having to wait for this function to execute in your script or your app you can send it over to celery and celery will run it well in your script or app you can continue on with whatever you were doing before so I'll give you an example of this and I'll show you how to save the result as well and get the result after it's been done and you'll see how easy this is all to get up and running it's only a few lines of code so the first thing is you have to install celery so I should have it installed already so pip install celery and in addition to celery you need some kind of task queue so I'm using something called a rabbit MQ and you can install this on your local machine normally I would run it on my local machine but it's not cooperating with me right now so I just created one on Heroku and you're going to get an amp URL so I'll just go ahead and copy bind and paste it into my code so we'll call this tasks dot pi and I'll just put it right there I'll move it of course but I'll put it right there so you need to task you and you can use rabbitmq you can use Redis but the end result is the same you know the task is put on the queue and then celery will read from the queue and execute the task so after you have celery installed you have a task you running then you want to setup your app to use celery so the first thing you want to do is you want to import celery so we'll say from celery import celery so lowercase C on the first one capital C for the class that you're actually importing and then you can create the objects so you can call it app you can give a name so in this case I'll call this tasks as the name and then you need to specify your broker so the broker is the URL here so if you were able to get rabbitmq working on your localhost then instead of having all this here you can just put the word localhost and it should be able to pick it up ok so I have that next I want to a task so the task is going to be the thing that I can actually run somewhere else so to do this use the decorator app that tasks and then you can define a function underneath so I'll define a function and I'll call this function reverse and it's going to take in some text and what I want to do here is simply return the reverse of this so to do that I'll return texts and then just reversing the direction of it so if I start up I open up another tab and I'll started with Python repple so from tasks import reverse and I need to start up the virtual environment in the second tab as well so from tasks import reverse and I'll just run it so reverse and I'll reverse my name and we see it returns Anthony reverse so that function is pretty straightforward and you know it doesn't do anything special but this is going to be our example and to make this even more clear what I'll do is I'll say from time import sleep and before I reverse that text I'll sleep for five seconds so now let me start up the Python repple and say from tasks import reverse and then I'll call reverse on my name again and it should take five seconds so this is kind of to simulate a long-running task you know it shouldn't take five seconds to reverse a stream but this is just for example purposes so I have that so this works fine if I call the function directly but what if I want to use celery it's that kind of offload this process and then continue doing something else so I'll go back to the original tab and what I'll do is I'll start up the celery service so to do that it's going to be celery - a capital A then the name of the file so tasks and then you have to say worker to start a celery worker and then you can say log level equals info so you can see what's going on so it's gonna start you should see your tasks there so I have one task called reverse and it says celery at my laptop is ready so now if I go back to the other tab and start a Python again and say from task import reverse now what I want to do is I want to send this over to the task queue so celery can execute it for me so to do this I have to call reverse dot delay instead of just reverse so by calling dot delay on it I will send it over to the whole celery process and then inside a delay I can pass them the arguments that I have so delay Anthony and it returns immediately and I have this async results if I go back to the other tab we see stuff happened here receive tasks tasks that reverse and we see tasks that reverse succeeded and then we see the result which is my name backwards so this is great because I'm able to easily sind celery any function that I decorate with app dot tasks and it will receive it and execute it and I see the result here but what if I actually want to get this result somewhere in my script or in my app well to do that I need something called a back-end so this is just a service that is just running and basically once it runs it kind of forgets what it did before so like I can see it here in the logs but in a sense celery has already forgotten that I've run this task so if I want to save the results of tasks and to save their status and everything what I need to do is I need to create a back-end now backend is just a place to store this so for this example I'll use a sequel Lite databases the backend but you can also use something like Redis for the backend as well just depends on your purposes but just like the broker it's just basically switching out some kind of configuration shrinks use the backend so the backend for this is going to be DB and then sequel lights and then I just need to specify a database name so I haven't created the database yet but I'll call it DB dot sequel like three and I'll stop the server so I can set up the database really quick so I'll do sequel a 3 DB sequel Lite 3 and if we look at the tables notice that it's empty but you'll see that when I run this through celery again it's gonna add something into the database so I will start up the worker and it tells me no module name sequel kameez so because I'm using the database as the backend it uses sequel alchemy to interface with the database so I'll go ahead and install that so pip install sequel came-- and then I'll try starting up celery again and it looks like it's okay this time so I'll go back to my other tab and what I'll do is I'll make sure that nothing interferes so I'm just restarting the repple again I'll say from tasks import reverse and then I can do reverse well actually I'll do result equals reverse dot delay and then I'll pass an Anthony and I'll get something in results and I can say result dot status and we see pending and do result dot status again so remember this takes five seconds and now it says success and if I want to get the value I do result dot gets and it returns the actual value so I can't get that value until five seconds has happened so I'll do this again and if I try result I get I get nothing it's gonna wait for the results to arrive which is you know five seconds later but if I check the status first and I see a success then I can call that get and I know I can get the result because it's already been calculated and if I you know exit out of here and open up my database again a sequel I three DB is equal like three look at the tables I see I have two celery tables now and if I do select star from celery task Mehta I see the two tasks that I ran and it tells me the time these question marks are basically blob so it stores the result inside of a blob I can't see it here through the secret like command-line but we know it exists because I was able to do that result back yet and I was able to get the value so if you ever want to go back and see the results in one of your tasks you can easily go in the database and get it from here so in that sense celery now remembers what it has done in the past so I know this was a really straightforward example but this is the basics of working with celery and it can be more complicated appealing on what you do but the celery setup itself is the part that is much more straightforward than the actual tasks that you want to execute in your app so I hope this has helped you if you have any questions about using celery and Python feel free to leave a comment now below if you like this video please give me a thumbs up and if you haven't subscribed to my channel already please subscribe so thank you for watching and I will talk to you next time you

Channel: Pretty Printed

Views: 48,877

Rating: 4.9484067 out of 5

Keywords: python, celery, asynchronous tasks, async, rabbitmq, message broker, celery distributed tasks, programming, tutorial, sqlite backend, celery backend

Id: THxCy-6EnQM

Length: 9min 6sec (546 seconds)

Published: Fri Jul 03 2020