How to create a great dev environment with Docker

this is a talk I recently gave at the docker Community All Hands event it's a follow along tutorial that teaches you everything you need to know to create an awesome local development environment with Docker so enjoy hi everyone I'm Patrick and welcome to my talk how to create a great local python Dev environment with Docker so let's have a quick look at what we talk about here first why Docker and then we learn about all the steps that are necessary to have a great developer experience with Docker so first we learn how we can dockerize an app so we start with the very Basics this is a very beginner friendly talk then we learn how we can get immediate file changes from a container to our system and vice versa so we talk about volumes here then we learn how we can set up an IDE inside a container then we learn about Docker compose and how this will simplify everything so then it only needs one command to start everything and if we understand stand how Docker compose works we can very easily add more services it only takes one more line for example to add a database then so I also show you this in a quick demo later and finally we also learn how we can debug python code inside a container so yeah I think these are all the codes necessary to set up an awesome environment and now a quick sentence is about a few sentences about me so my name is Patrick lober I'm a developer Advocate at assembly AI assembly AI is a deep learning company that provides state-of-the-art AI models as an API and here we use Docker for pretty much everything so not only for deployment but also for local development so we're big fans of Docker also I'm running my own YouTube channel so you can find me with the handle at part lower and the same on Twitter pad lober and then I'm also the founder of the Python engineer website so check that out as well for Python and ml tutorials but now enough of me so let's get started and first why Docker and why not just use a virtual environment so Docker is much more than a virtual environment it's basically a whole Linux operating system that is a isolated environment where you can then install all your dependent sees so not only the python modules but also all other dependencies on the system and you can keep your own system clean and don't pollute this then you can very easily add multiple Services we will learn how to do this later you can easily deploy this to the cloud then your co-workers can have the same environment by spinning up the same containers so also very cool to reuse um the environment and you can also quickly test out different python versions so for example when the latest 311 came out in October I use Docker to pull the latest image from the docker Hub and then quickly and spin this up and test this so yeah I think super cool to play around with different python versions very quickly and now before we start I also want to mention that there is another Docker feature that is called Stef environment so this is currently in beta I think and this should come soon so this is not what this talk is about in this talk we learn how we can set this up step by step together so that you learn all the docker commands and the docker features yourself so you understand what's going on so yeah but yeah I think this will be a cool feature that's coming soon as well and also now since we do this together everything step by step I also want to mention that all the code and the commands that I use they will Avail will be available on GitHub so you can find this here GitHub Patrick lover python Docker tutorial this is the repository um but now let's get started with a demo app and as a demo app I want to use or create a small fast API web application so for this let's check out the documentation and the first thing we would normally normally do is install the dependencies with here pip install fast API and pip install uvicorn but in our case I don't want to do this on my machine here I want to do this in the docker container later so I want to save the requirements in a requirements Dot txt file here we put fast API and UV corn and now let's create the code and for this I recommend to create a new folder that I like to call source and then in here let's create our main dot Pi file and let's grab the example code from the website so this is all that we need in order to run our first application and now we want to dockerize this so for this we have to create a a file that is called Docker file in the base directory and here let me quickly copy all the commands and then let's go over them together so this this this this this and this so first we start with a base image and usually if you have a python development environment you often start with a python based image and then here you can specify different tags and this is where you can very easily switch out the version for example here you can use 311 I want to use 310 and I also often like to use the slim version this will just give you a much smaller version than the regular one regular one but it still works fine for development now the next thing I want to do is to create a working directory that I call code this will now be the starting directory inside the container so later you will see what this means and now first let's copy the requirements file and then install this and here we use dash dash no cache term because Docker has its own cache so we don't want this here and then we copy the code so the whole Source directory and then we specify the entry command and it's good practice to have this separate because Docker can cache each step and the requirements don't change that often so now if we have changes in our source code then it can start building this from here for example and now to the entry command so if we have a look at the documentation then here in the terminal we would say ubicorn main app dash dash reload and here essentially we have the same but here we need this as a list of strings so we have ubicorn main app and dash dash reload and be careful since we are inside the source directory here we also need to say Source main colon app and then here we are also already specifying the host and the port so for the docker you usually always use zero zero zero zero and then you can access this from your localhost as well on Port 80. and this is all that we need in order to build our image so now we can build our Docker image and for this we say Docker builds minus t and then we give this a name fast API image and a DOT for the current directory and now this will build the image so in my case this was very fast because I did this before so it was cached if you do this the first time it will take a few moments longer but now we have this so we can check Docker images and here you should see your fast API image so now the next thing is to run this and we do this with Docker run and then let's give this a name one second let me shorten this here a little bit so this is the first command we want to use Docker run dash dash name then we call this fast API container then we want to map the port from 80 to 80 and then the image name is called Fast API image so now we run this and now in the terminal you should see now this is starting so this is using this starting command and now since we mapped the port from 80 to 80 we can now access this on our local machine so if we visit localhost Port 80 we now see the app up and running and here we get the response so this is working so now we dockerized the image but now let me show you um one thing that I want to do differently here so let's stop this again with Ctrl C and let me actually also say Docker remove fast API container otherwise I think it will complain and now we use the same command again Docker run but here we also say minus D and this means it's in detached mode so now as you will see it will immediately give me back the terminal and it's also showing this ID here so now this is running in the background on my machine so I can still visit the local host and test this and also if we check Docker PS then you see the container is still running so let's quickly go to the localhost and refresh this and it's still here so now this is running on the background so now we have um but now we have one a few more problems I would say so first of all if we go ahead and change something in the code so let's return one two three four then even though we say minus minus reload if we go to our localhost and refresh this we don't get the new code because um this one here the code is on my on my local machine but the app is running inside the container so the next thing we want to do is to have a mapping from our machine to the container that Maps the code so we can quickly make changes here and then it will affect the container so for this let's stop the fast API container and also let me remove this again and for this we want to use a volume so if we have a look at the docker documentation volumes are the preferred mechanism for persisting data so in order to use this we can do this with an additional command so or with the same command but an additional parameter so here we say Docker run and now in the end we say or somewhere in here you say minus V for volume and then we want to map a a location from our machine to the location inside the container so I use um dollar and then PVD this is the current working directory and I want to use to map this to slash code so this is what we are using here and if we for example open a new console and type PVD then you see this will give me the current working directory where the code here lives so now if we run this again and now we have this as a volume then now it's starting this again in the background and now we can visit our server again so let's refresh this here we have hello world one two three four and now if we go into the code and make a change so let's say hello world exclamation mark and save this and now if we refresh this then now here this is updated immediately so this makes our life much easier when we develop here but now we have another different problem so we still can't really code in here because my IDE is already complaining about fast API for example if we want to go to the definition then it says no definition found for fast API so we don't get Auto completion and all the good stuff and this is because my system my my IDE here doesn't know about fast API because I didn't install this here I only install this inside the container so what we actually want is to have this IDE inside the container and don't worry this is not too difficult to set up you can do this with both vs code and pycharm very easily in my case I use vs code and for this we want two extensions the first one is the docker container Docker extension this one and the second one is called Stef containers so install them and then in the lower left here you should see this little button here and then there are different ways how you can connect to a container for example we can attach to a running container or we can open a new folder in a container if we have a Docker file in this container in our case since we have our container running so it's still running in the background we can attach to a running container and then we select this one the fast API container and now this opens a whole new instance of vs code and this lives now inside the container or basically it's just connected remotely but yeah it's living inside the container and here as you see it's no longer complaining about fast API so here it should know about this but we still and for example if we do a right click we don't get all the other suggestions like go to the definition and this is because this is a whole new environment so here we again have to install the python extension first and don't worry you only have to do this the first time so here let's install this in the container and this might take a few moments so oh no there's already here perfect so now we have this and now if we go to this and then we can go to the fast API definition and then jump around in the code and also we should get Auto completions so yeah perfect so now we can develop inside the container and also remember since we did the mapping with the volume also this works the other way around so for example if we do some changes in here and then let's go back and refresh this then we also see this here perfect so now we can code in here and Implement our app and we get all the support from the IDE that we are used to from our machine we even can customize this even more with other extensions or install other dependencies and make this a completely new isolated coding environment that has the python 310 install here that we specified here so you can easily switch this out we also immediately see the changes and can test this and we also don't lose the changes so since we used the volume we now immediately have this on our system as well and can then push this to GitHub then for example if we start a new terminal you will see that this will be inside the container now and it is inside the code folder so this is the working directory that we specified here for example if we see the one level up and list everything then you see all your Linux folders here and also the code code folder so let's see the into code again so this is why we specified this year as a worker and here if we list we list we see the requirements and the source so this is what we copy here and since we are using a volume that Maps everything right now we also see those two files and yeah so this is now how we can set up an IDE inside a container and now I want to talk about Docker compose so um let me stop everything so let's close the remote connection and this instance and here we are in the old one so as you have seen we use this very long commands with the minus p with the minus t with the minus V everything so now we can simplify everything with Docker compose so let's have a quick look at the documentation so compose is a tool for defining and running multi-container Docker applications so here we Define one yaml file and in there we can configure all the services and for example then also Define the parts and the volumes and other stuff and then we only need one command to start and stop everything and we can easily add more services so let's do this let's actually stop this again so we no longer want this so we stop the fast API container and let me also remove this and have a quick look so now we no longer have any containers running and now instead of the docker file or rather in addition we also want a Docker compose dot yaml file and here let me also copy this from the readme and then we go over this together so first let's use only this and here we specify all the services that we want so for now we only have one that we call app then we specify that we want to build from the local Docker file here we can also say in image and then specify an image name from the docker Hub and you will see how to do this in a few moments for now let's keep the build then we give this a container name and then we give this a command and essentially this is the same commands that we are seeing here but now written in the terminal Style again and then here we have the configuration for the parts and for the volumes and essentially this is the same configuration that we see here in this long command with this minus p and the minus V so this is just a different way to configure this and now instead of this whole command here we only need to say Docker compose up and now this will start all the services here so now as you can see it's running again so we can visit UV our localhost Port 80 and here we have this and we can also make changes and then we should immediately see this so in the terminal you see we have some more infos and if we refresh the page then we immediately see this so this is working and now let me stop this again and in order to shut everything down correctly we want to say Docker compose down and now I want to show you how we can add more services very easily in here just by adding a few more lines so in this example let's add redis for example as a database or as a cache so we Define a new service name and now instead of the build command here we say image and then a image name that we can find on the docker hub so let's search for redis Docker Hub and then we should immediately find this and here you also see the available text so I want to use redis Alpine so let's copy this in here and now for our first service we want to save this depends on the redis service and this is all that we need in order to add multiple services to the docker compose and here of course we can also then specify more configuration but this is all that we need in order to get this and now this will pull red is from the docker Hub and then also install this inside the container then we don't have to install this on our machine and now before we restart everything let me also add redis as a python dependency so this is the redis python module and now we can use this in the code and now we want to say Docker compose up again and this time let's also say minus minus build because we changed something in here and when we want to rebuild our image so let's say minus minus build and then let's also say minus D so now it's will it will again be in detached mode so let's run this and the build is done so now for example if we say Docker PS then we should see the python server and also our redis image so these are now both running in the background and now we can for example add redis here to our code so for this let's also attach this again so let's say attach to running container then we want our python server and now it's restarting the vs code and now I think this will be again a different one so I think we again have to configure python the python extension here don't worry you only have to do this once and then it will always be there but since we used the docker compose file now for the first time this is building a separate image and a separate container now down so but now we have this and now we can use this so we should be able to say import redis and it should be there with auto completion so I think python is still installing so let's wait until this is installed and now installation is done so let's try this again import redis and now you see we have the auto completion so it finds redis so now let's quickly add this here so for this we say R equals redis Dot redis and then as a host we say host equals and this is our service name which is also called redis and then for the port we use the the default Port which is six three seven nine and now let's for example Define another endpoints where we say slash hits and then here we can say r dot increment and let's give this a key so let's call this hits and now here we want to return hello r dot gets and then we say hits and here we say the number of hits and now this should be working so everything is already synchronized so let's refresh this and let's go to slash hits and here we see the number of hits one and if we refresh this then we get two and let's refresh this again three so our redis service is working and this is how we can easily add more services to the docker compose with just a few lines of code and everything is still synchronized with the one Docker compose up command so yeah I think this is a super cool feature and now the last thing I want to show you is how we can debug the code inside our container so if you go in here and for example set a breakpoint here then we want to be able to start the debugging here so for this let's first close this again and also close this instance so for this first we need to add one more package to the requirements txt and this is called debug pie and then we also need to add the port of debug pi to this part so it's five six seven eight and we map this to five six seven eight and now we can again say Docker compose up minus minus builds minus D and run this in detached mode so now this is building again and now we have the debug Pi package then so let's again connect this to the running container and now this time we should be able to have our python extension again and now we should be able to for example go to the definition so yeah our python extension is working and now in order to debug this we need to add a small code snippet so we say import debug pi and then debug pi and you see it's already recognizing this and then we say listen and then here we use a tupler and the end point will be at zero zero zero zero and the part is five six seven eight so this is what we specified here and now we could also same debug Pi wait for client and then it's blocking here until we connect but here we can let this run through and now when we hit the end points then it should trigger this break point here so now this is all the code snippet we need and now we can attach the debugger so let's start run and debug then let's say create a launch Json config and here we can select remote attach attached to a remote debug server this is now where we are running so we can leave the localhost since we met this and also the default part is five six seven eight so let's hit again enter and now we created our configuration so now our debugger is running and now if we go to the endpoint slash hits then we should end in the debugger but first of course we have to also start debugging so let's start this again and let's hit the end points again and now we are inside the debugger and it stopped here so now we can inspect all the variables and have our debugging up and running and then of course we can step over this and then just like with a normal debugger and do some more sophisticated debugging the other option that I didn't mention I forgot to mention of course we can always go to our container and then have a look at the locks and then do for example some debugging with print statements so for this yeah I have a look at the container that you find in the docker GUI but this one is a little bit more advanced and it's also not too hard to set this up so yeah these were all the things I wanted to show you in this talk I hope you enjoyed this and could follow me here if you have any questions drop them in the comments or feel free to reach out on Twitter also check out my YouTube channel and yeah thanks for watching and then maybe I see you next time bye

Channel: Patrick Loeber

Views: 125,360

Keywords: Python

Id: 0H2miBK_gAk

Length: 29min 54sec (1794 seconds)

Published: Wed Jan 04 2023