Docker - Django, Celery & Redis Docker Compose setup

hello and welcome back to another docker series tutorial so this one is a walkthrough on how to set up a docker compose file for django salary redis and postgres database so i am making a few assumptions here that you know a little bit about django celery how to separate set up a simple worker for example i will go through it we'll go through every single step in this tutorial but i might not give you the level of detail that you're expecting about everything here so the core of this tutorial is the docker compose file because that really is the glue that's going to stick everything together to build this environment so you can work within this django celery and redis postgres environment so this is what we're going to end up with containers we've got the redis container the postgres database django and celery container and that's all working nicely together to then you can see here generated or run a task and we've just added some simple numbers together and we have an output of four so that's just showing that everything is working so that's what we're working towards so let's get going so first up let's just create a new project i'm going to call this core and put it into this new folder called django so next up we'll just go ahead and start a new app i'll just call this app there we go so we've got an app there and that's about it for the actual django setup for now so next up let's build a new file this is going to be a docker file so we'll start with a new docker file inside of here we're just going to define the fact that we're going to be utilizing a python container so i already have docker installed i've got some images already downloaded so the python image uh postgres and redis so that's all ready to go if you don't have this downloaded it's okay once we actually build this through docker compose they will just get downloaded of course so inside docker compose we're going to need to set up or define the fact that we want to use python i want to use python 3. we can set some environment configurations so for example we can go ahead and utilize the python unbuffered equals one so this is basically going to allow us to log messages immediately um instead of it being buffered so let's go ahead now and think about the next one so we need a working directory so a setup a working directory um it's going to be in the user src and then app so that's going to be our working directory in the container so we want to have some applications installed so we go ahead and run the copy so here we're going to build a requirements text with all the applications or dependencies that we're going to need for this and then finally we want to then install them so there we can just run a run uh pip install uh our requirements and we're going to run the requirements text so let's go ahead now and create a requirements text for this so we're going to want to run django we're going to be utilizing a postgres database so the psy cop g2 binary we're we're gonna run salary and redis so let's go ahead and create a requirements text and inside of here we're just gonna define four items so i'll just paste this in so we've got django here i'm just defining uh the latest version between 3.0 and 4 and this binary here um for the connectivity for a postgres database um obviously we're going to be utilizing celery and reddish so obviously we can just go ahead and just select the latest version so we don't have to define the actual values of course um so now that's in place we have that and that will be ready to install so let's not forget this docker file is a text document so we don't use a file extension here in case you're wondering this contains all the commands a user could call on the command line to assemble the image so this is this is basically the command that we're going to utilize to assemble the image obviously we create the containers from the image so now we know we're going to build a python container eventually um it's going to have the requirements already installed so our container is going to have the django package installed celery and redis so that's all good to go so let's go ahead now and build a docker compose file so this is going to be a yml file um or yaml file so we first of all we're going to define the version so the latest version as of developing this tutorial is 3.8 and now we need to find some services so we're going to need a few different services here so let's go ahead and complete this now so first of all we're going to define django and then what we're going to need to do now is define the build so in this case our build is dot this is just referring to the fact that our build context is in the same directory here so next up we have the container name so i've just named the container django so that we can easily refer to it so most of the times you might want to use the commands to you might want to run command sorry utilizing the id of the container by defining a name it can be a little bit easier to apply commands to a name than it is to an id which can be fairly long so now we have a command so this command will be run for example when we create the container we want to ensure that our django server is running so we run the python manage run server and we run that on the zero zero zero by defining the quad zeros here we mean listen on all interfaces and so we will listen or the django server here will listen for connections on all ip addresses so next up we define a working volume for our django application which of course will be milled across to the container so we can work here inside of our visual studio code editor any changes that we make here will be changed on the actual container as well so then we define some ports so we bind our host 8000 port to django so we're going to be utilizing the 8000 port just the default port that django utilizes when we start of course you can change that and then we're going to set up some environment configurations so what we're going to need we don't need to do it what we can do is set debug to one we can set up the allowed hosts so we don't need to go in and configure the django application so this will just allow us to run or access our django application from the loopback address 127 then we're going to set up our salary broker so this is just a a defined environment parameter we set up so redis is going to be utilizing 6379 that's the default port of course again we can change that so have a look at the uh documentation for these additional parameters parameters here defining the database and then potentially also then we can define the the the cache as well at the end here so next up we have the back end so celery back end we're going to be utilizing reddit so we've defined the broker and the back end as redis and then we're going to say this depends on so not in it not only do we want to run django but we also have other services that this relies on so we're going to need to set those up so we're going to set up a postgres database we're not actually going to configure it in this tutorial we're just going to show you a configuration if you wanted to use postgres database so now we have also another dependency of redis so we're going to rely or we're going to also build a redis service so let's go ahead now and define our salary service so here we defined a salary service so we could we don't need to necessarily do this we could just run salary directly from this service here but we've just broken this service down into a separate service we're going to be utilizing the same build context and we're going to run the command so this will just get salary started when we actually run the docker com docking compose file so that's just going to get celery started um when we run when we fire this all up and we can also define the same volume um so we're usualizing this volume here so we can make changes and then this depends on the django service and also redis okay so the next service is a database service so we're going to use the image of the the progress image so like i said before i've already downloaded this it's already ready if you don't have this it will just get automatically downloaded okay so this is the image i'm going to be utilizing for this site also name it pgdb again just a convenient name for me to access this service i can then set up some environment variables so you can follow the um if you go over to the docker hub just have a look at some of the environment configurations that you can apply so here i've just created a a database called postgres user postgres password postgres and here i'm going to set up a volume a named volume so i've got my name volume here pg data so this just allows me to store the database data in a named volume if you're not too sure what volumes are check out the previous tutorials and then i can go ahead and now set up my redis service so with redis all i need to do here is define all i've done here is to define the version of redis i want to use so alpine is just a a slimline linux package that redis is going to run on top of so this again is just an optional thing have a look at the docker hub for the different options that are available here so finally i'm just going to define the volume because i've set up a volume reference here so i just needed to find that volume and we're ready to go so that's pretty much it for the configuration we've configured the database with a volume persistent volume we've been we've got redis um defined here uh so with redis i think the um the image comes uh pre when you run the container redis will just start automatically so we've got celery here and we've also defined the django service which is going to be work run on top of the python image okay so just to get things working once this service is started um i've created the app folder here if you remember so let's just make sure that we've added that into the installed apps and then what we're going to need to do also is just set some more configuration option options here so that we can connect to our uh redis service so we want to configure two options one is the salary broker url and so the next one here is going to be the celery result back end so we can set up these two configurations right here now you notice that os dependency is missing so let's just go ahead and just import os okay so now that's in place right so that just connects everything up nicely with the um the redis package or service sorry um so what i need to do now is i need to make a new file here called celery so let's just define a really simple celery instance so please head over to the celery tutorial series in this channel if you want to learn a little bit more about celery but i'll just breeze through this so we've got some imports here we're just setting up our our salary parameters so that we can run salary and just run a simple test or a simple service so let's go ahead and obviously we're going to be utilizing the core settings um so we bring in core and then we set out auto discovery tasks it just automatically finds any tasks that we have in our application so we're not going to define them individually and that's about it so we've set up celery there so let's just go over into the apps and let's just create a new file here called tasks so we're just going to set out a really simple task here so we want to run from salary import shared tasks because we want to run a shared task and then we're just going to run a simple addition task so we've just got two numbers that we're going to take in and then we're going to add them up and return them and then finally in the the init file here in the core just allow django to find this all automatically so we add some code there right so that's pretty much all it that's pretty much it what we need to configure just to get this working in a really simple standard way right so let's go ahead now and actually run this so let's go ahead and fire this up so docker docker compose run and then what we want to do is define what we want to run so in this case we want to run django so let's do that so that would take a couple of seconds to do that okay so once you've done i'm just going to close that turn this all back on so docker compose and up so i just wanted to show you this so once you build this you can see that all these services are being turned on here and we've got the redis server and the postgres database we've got celery here um it says error no such option app okay so just look in the sorry in the requirements file i didn't actually specify the version of celery so this is going to be utilizing the latest version which is as of recording celery 5.0 so just go back into the docker compose file here um the command to actually run celery is slightly different so we're going to use the a switch here to find the name in this case core so the app equals celery core um and then run the worker and then we're going to use the l switch or flag and then info you have to be really careful here because for example in the documentation it will have a lowercase info but that will cause an error it needs to be um in capitalized it needs to be capitalized sorry um so that should run the worker now right so let's go ahead and try this out so let's just go ahead and uh rebuild uh close that control c and then i'll do a compose up so i've just closed that and then compose up and that should just take us through the process and show us everything coming up here we go uh so there we go so if i scroll to the top here you can see that redis is um started or sorry django's started running on the 8000 port and we have redis um and then further down here redis is now set up we've got a database set up so that's now configured and redis comes in again and down here we we have celery so salary has been started it's running you can see the tasks that are available are the tasks add and that's obviously the simple task that we created here in the task file if i can just get that there we go so everything is running nicely um it says here that for example it defines the back end um so we've got the app the transport and the results it's all defining redis so that looks like it's all working if it couldn't find the back end it would tell you if there was a problem between celery and redis then celery would output an error so everything looks okay so let's go back into the command prompt and just test this out let's open up a new command prompt sorry and let's go into the django instance so docker exec and then we're just going to name it django so we're not using the container id just to contain name because that's what we've named it if we go into the containers we've named it django okay so let's go into the shell and now let's just go into the python shell so this will enable us to then actually run um a salary task so let's just import the task first so from the app um dot tasks import and we want to import add so now we have that we can now run a simple task so let's run add delay and then two and two so here we just um ascending across the two parameters two and two and then we're just obviously going to add them up and that's going to be the result so if we fire that off you can see that we've got an async result if we go back into the docker compose here and just bring up the celery at the bottom here you can see that we've it's received the um the task it was successful and the end result was four two plus two is four so clearly everything is working nicely and there we have it docker compose django salary postgres database and redis

Channel: Very Academy

Views: 15,609

Keywords: django, django celery, django docker, docker celery, django tutorial, djangotutorial, django tut, django 3, django examples, learn django, django framework, django 2020, django example, celery django, django beginners, beginners django, celery, django celery example, docker volumes, docker container, docker, docker windows, learn docker, docker tutorial, docker compose

Id: oBQxFn1CDno

Length: 20min 29sec (1229 seconds)

Published: Tue Oct 13 2020

Reddit Comments

[u/akuani]

This guy needs more views.

[u/OMDB-PiLoT]

Doesn't Rabbitmq outperform redis as message broker? Are there any recent benchmarks available between the two? Especially when handling large number of msgs?

[u/ShiddyFardyPardy]

Ssssshhhhh we may have to start paying for this guys videos if he gets too popular.

[u/egehurturk]

The channel itself is pretty good. I like his advanced django tutorials

[u/Next_Concert1122]

please make a video on CI/CD with gitlab using docker

[u/Mandelvolt]

!remindme 1 day

[u/memo_mar]

looooving it!