Deploying Your Python Applications

and good morning welcome from texas on this moderately cloudy overcast day little gloomy been like this for a while now just kinda been raining it's been a really wet year really wet year for texas this far not used to it uh just giving everyone a little bit of time to hop on to the live stream before i get started today we're going to be talking about deploying python applications um if you check out the tech talk page at do.co tech talks i believe um and i'm sure a link will appear for it in the comments momentarily uh i've listed a lot of resources in that tech talk page where that i'm gonna be covering some of which but everything's gonna be covered is pretty much coming straight out of a tutorial oh hi from austria hey get profit how's it going it's nice to see everyone get get profits here jimmy's here nice to see everyone this morning uh but yeah i'm gonna be talking about deploying python applications so if you've been keeping up with my tech talks for the past oh i don't know like the last four i've done a whole bunch of python coding tech talks um and we haven't talked about deploying them at all and today we're going to talk about deploying them right from mexico hi jimmy javid says hi hi from rio we've got romania uh says hi sas i'm sorry sasidharan i'm trying david says hi belly says hello hello from russia from heinrich it's awesome poland uh please use environment variables for passwords we'll do the best we can um yeah we definitely will be using environmentals for passwords hello from from mexico from london from brooklyn new york we've got a lot of people today sri lanka namaste to you as well hello from london we got oh there's so many people coming in ecuador wow they're like never realize how global of an audience i have it's awesome colombia martin says hello from argentina i didn't get that one long time nicholas is from the philippines vekas is from india wow they're coming in so fast eduardo's from brazil yasin says hello george from uk it's good to see everyone turkey portugal argentina india ecuador estonia okay we've got some estonia here i'm clicking on everything good to see everyone again too and tune army captain the buddy of mine from wet texas yes it's been very wet here this year from brazil uh yes it's been a wet year i don't i don't understand it it's a little bit odd hello from india sri lanka kenya awesome hope to visit a lot of these places when i like traveling i'm a traveler i haven't really got the chance to be much of a traveler because of covid i joined it as a developer advocate to get to travel and then immediately got thrown into a pandemic so uh for everyone out there i hope you're staying safe from all this this year from nigeria ooh saudi arabia lots of really cool places it's awesome to see everybody ooh bangladesh i love seeing where everyone's from it's one of my favorite things it's so cool some of you are up pretty late nope i miss this one from germany up here australia yay love australia i'm gonna go one day i want to go to pycon australia i've been wanting to go i was planning on trying to go and then of course pandemic uh how are you doing i'm doing well thank you for asking we'll give it another we'll give it another minute we're going to start momentarily i promise i'm just letting everyone kind of get a chance to come in i think we're getting good numbers on the stream so far i can see like the viewers so i think we're going to go ahead and yeah let's go and get started now we're not going to give another minute because today we got a lot to cover um okay so if you've been watching my tech talks for the past two months probably i think it's been like all of q2 i've been talking a lot about python frameworks we've covered flask we covered django we covered the django rest framework we covered uh fast api so we covered a lot of stuff um and i forgot about that oh i haven't deployed any of them we've mostly just been talking about that but today we're not going to write any python code we're going to do nothing but deploy our applications so what we're going to do today is we're going to deploy we're going to build and deploy basically three types of applications we're going to build a wsgi uh we're going to build a server which is my tech talk droplet right here that's going to deploy our django server using wsgi and it's going to use a url shortener piece of code or a url shortener code which is in bo community i'll post it in the chat um and this is a just a basic django app kind of acts as like a little bitly um where it basically just creates a url shortener and it does a redirect for you real relatively straightforward but it uses django and uses postgres we're going to do this on a server then from there we're going to take a really tiny flask application that i wrote uh called website redirect and if you look at the code for this it is literally just a redirect it's an it's a flask redirect this is if i want to take a dns name and redirect it to a kind of thing it's like bitly but it does a one-off and it just does one thing and it just kind of runs there so i use this one personally running for discord.mason.dev which is how you can join my public discord server if you're interested and it just does a redirect this is something that i could have done with say like nginx or apache and just use like a basic web server but it's a really simple code we're going to take this code and we're going to build a docker container and we're going to run it in a docker container locally upload it to digitalocean's container registry where we're going to use it to deploy it to app platform it could also be deployed to kubernetes if i get time i will try to do kubernetes deployment um but it's not going to be like a cube cuddle as i'm going to use the kubernetes uh like dashboard to do it so um doing python on kubernetes is a big talk and if you're interested in that talk leave a comment in the chat i would love uh to talk to you about like if you want to hear about python and kubernetes say that's what you want to hear and in the chat we'll tally it up and if we get enough people that want to hear about it we'll talk about it lots of stuff don't have time to cover all of kubernetes today and then the last thing we're going to cover is the one-time secret app that we built if you were here for my last talk which was fast api we built a one-time secret app that works really similar to just onetimesecret.com which basically you put in a secret you put in a passphrase and it creates a link and if you take this link and you put it here and you type in you can see the secret okay this is basically just an api for this um but we are uh we're going to build this on fast api this we're going to deploy directly to digi lotion's app platform so as you can see we've got a lot of stuff to cover today so let's get started i definitely am seeing people uh saying that they're interested in the cube stuff great now i have to learn kubernetes or relearn kubernetes it's been a while okay so what we're going to start out with is with our url shortener and we're going to deploy to a droplet so we have a droplet that i've already pre-seeded with some stuff so basically what i've already done is i've already done an app update and i've installed nginx certbot virtual environment i've installed my development tools and the tools that i need to deploy my python application so the first thing we're going to do in that because i haven't done it yet is i'm going to add a dns record because i want to keep this secure using the uh https so i'm going to need a dns record so we're going to call this oh we'll call it tech talk at sami.cloud yeah that works we'll call it tech talk droplet we're going to create that record and we're good so we come back here let's go ahead and grab our ip address ssh root app talk dot sammy dot cloud and we're good uh i'm i normally i would set up a direct login into my non-root user um but we're not so we're gonna be doing all of this from a non-root user because it is a good idea to not deploy code as root it's always good to like limit the privileges as little as possible um and yeah so what we're gonna do is we're gonna go directly into this and first but first we're going to talk about how you deploy on a droplet because droplet deploy the old way you're not the old way i mean it's kind of the old way when you think about like paths and kubernetes and stuff but the the server deploy the vm deploy is very much dependent on you know deploying with a web server and using something called wsgi so if you've never heard of wsgi it's a great site for it that i use that pretty much full stack python explains it really well um your traditional web servers like nginx and apache don't natively know how to run python code you know these are all like cgi bin based uh web servers and you need some sort of thing to run it so we have the wsgi which stands for the web server gateway interface and it's basically like a middleware layer of code that allows you to basically take a gateway and invoke a callable object to it so the purpose of it is it basically the web server runs this other small application that stands up you know kind of like a web server um and it can set up multiple workers which is really good and it helps with the flexibility of this um it allows you to you know a little actually the flexibility you can change out the whiz keys and not have to really worry about it they work they're all uh compatible with what's known as pep3333 if you don't know what a pep is it's a python enhancement proposal um but it's basically the things that uh are written to add things to python so it basically just acts as kind of like a little like server that manages your python apps and then the web server proxies all of the data from the web server to the whizzy it does the stuff and returns back um it really helps with scaling because as we all know python is plagued with the global interpreter lock um which allows for doesn't really allow for the greatest of multi-processing experience um so if you want to be able to run your python on more than one core you need the wsgi because it basically creates more processes and runs it kind of like little workers in the background um there are a lot of different wsgis uh the one there's like g unicorn u whiz gee which is pretty popular mod wsgi g event you could even get away with just using there's a lot of them today we're going to focus explicitly on g unicorn it's the one that i know it's the one that i've been using um you whiskey and mod whiskey are great i used you whiskey um i think at one point like at the very beginning of my career it was really one of them i don't remember i think it was usually it's really bulky there's a lot of really like it's got a lot of plug-ins and stuff um and i just prefer g unicorn so we are going to go ahead and just go with that so without any further ado let's let's deploy a python app so the first thing i want to do is i want to clone my code i need to get my code from github so we're doing the url shortener api first yes we are so we're going to go ahead and just copy clone it hit clone and let's get our environment set up so url shortener api we're going to do python dash mvn vn we're basically going to create a virtual environment here alias python equals python3 because i'm lazy so we're going to have our requirements here and we're going to go ahead and set up our virtual environment so we've got our vm let's activate it um source vn bin activate and let's go ahead and install our requirements.txt and while we're waiting on our requirements to install i'm going to answer this question can you explain the difference between wsgi and asgi yes so wsgi is the older one um and it's gonna do this because i don't have a wheel installed but then it's just gonna build the wheel so don't worry about the red um so wsgi is is an older one it's just the websocket gateway interface their web server gate i always say sock i think it's web server gateway interface asgi is the asynchronous server gateway interface it's whenever python added the asynchronous ability um python36 i believe is when asynchronicity really took hold i think it was early like it was in earlier but i think like three three six is just like a monument in the python space so like you're gonna see things that are getting deprecated and they do deprecate from three five back three six is really like monumental uh python three um and the asynchronicity basically is just a different way of doing it it's a way of running a web of your web server but if you're using async calls like async and await which are things that make your python code faster if you've written an asynchronous web server you use an asgi if you've written a synchronous web server which is what django uh is by default you can use asynchronicity in django but like if you've just written like a typical django app and you've not done anything with like async and await and all of that you use a wsgi we will deploy an asgi at the end that will be done on app platform because fast api is an asgi it's an asynchronous um python framework i'm not positive i'm going to say i guess i'm pretty sure that django supports asynchronicity and like you can do it but i think out of the box default it's a synchronous server using the wsgi so great question i'm glad you asked that we're going to totally cover both wsgis and asgis today so we've done that we've got everything installed um we have our database ready so over here i've pre-created a lot of things because sometimes they take a little bit of time so we have a postgres database uh yes celery is another asynchronous one that you can use um yes celery is a great option for async that's kind of like yeah you know i know of celery i've never actually used it so um it's a little bit older but yes celery definitely does async someone asks do we really need the web server nginx or can we use wsgi only because i oh i need one docker image to run the application when we get to doing the docker deploy we're not going to have nginx we're just going to do the wsgi so um we're going to do it a lot here so like this one we're going to use nginx because we're deploying on a server when we build our docker container which is the next part of this tech talk today we're going to remove the docker container and just use that so you're getting a little bit ahead of me but great question um asgi is really kind of used for everything like you can use it you like we use it you can use it for stuff it's good for chat applications as collins points out but it can be used for a lot um depends on how i think it really does help with um i can't think asynchronously ah it helps it just quick speed gets you a lot of speed so we have tech talk post for here for our postgres database and what we're going to do is um what was i doing i'm going to get the connection string right yes database url so i'm going to go ahead and we're going to do this over here and get the show password so you don't like i don't mind if you see it because i'm going to delete this database and as soon as i'm done with this tech talk but i'm going to pretend like i'm being secure um and actually yeah we're not it doesn't even matter because i'm about to do it on the screen anyway anyway don't don't be mean and still my database this will be gone in a minute anyway uh database underscore url equals and then we're just gonna put the database here we're gonna clear that so um what we've done is we exported the database url so that way our django app is built to look for that that environment variable and then um run it so now that we have that we need to set up our django site so like there's always the original like things you have to do for django and oh i want to do i want to do one other thing real quick uh we'll do it in a second um so we're gonna do python manage.py migrate and that's going to do our initial migration so now this is the thing you have to do whenever you originally set up a django site it has to you know prep the database get the tables added and everything we only have to do it once so if you were to add more nodes to this and stuff you don't have to do this every time we're going to do uh python create super user i have to do python manage dot py create super user we're going to create a super user for myself i missed up no i got it and then the one thing that you have to do every time depending on your site is you have to do collect static we have to pull in the static files so that way um we have a working static file directory for us so we've pulled in all the static files and um this will allow us to basically like now we have a static directory here and it serves like our admin stuff the rest framework stuff all of the static files work so what there's many ways you can deploy a django app on python using nginx we're going to do it by creating a g unicorn websocket and then we're going to create a systemd service that will allow us to basically turn the socket on or off and then nginx is going to proxy back to the g unicorn service so the first thing we're going to do is we're going to uh with the pseudo vim etsy system the system g unicorn dot socket sockets are the socket interface or the socket creation is actually relatively straightforward um it allows us to we basically give it a unit we give the description we're saying hey what do we want to name the socket listen the socket on run slash g unicorn dot sock so we're going to create a socket we're going to create name it g unicorn.socket or sock and then wanted by sockets.target this allows us to basically make sure we don't start up out of order systemd does have some um notorious like you can get things in race conditions we want to make sure that we um that we're ready to go for that and that way we don't start before everything else is ready because you can call you can get into some weird race conditions with systemd so we've created the socket file and in the socket file we'll run um but the next thing we need to do is we need to create a systemd file so systemdgunicorn.sock.service this one's a little bit more complicated so first thing we're going to do and i'm actually going to copy paste and we're going to talk about it because i've already gotten this to work and i'll be as you can see there's a lot to it so let's talk about it line by line unit we're giving what do we want to name this unit description it is the g unicorn daemon if you ask like what it is that's just a description it's like a name requires what do we require for this to turn on we require that g unicorn.socket has started before we start our g unicorn service we're going to say after network.target this is the network to come up there's no point for us to start this until the network comes up and again creates nasty raid conditions race conditions so when the network is up go you're good also by the way everything i'm talking about is in a tutorial in the digital ocean in my tech talk like and there's resources you can do this line for line like i guarantee you i did not know this off the top of my head i had to read a tutorial to remember how to do this so in the digitalocean tech talk page that for this exact talk there's at the bottom there's resources there's a tutorial that tells you all of this so if you're missing it um like or you think i'm like you know you can always go back and read it it's already written so the next thing we need to do or let's go down to install install we wanted by multi-user target this is kind of just the um this is kind of the like what install and want to buy is like when do we start multi-user targets kind of like the one of the last things that get started like whenever all the system stuff is coming up i don't remember exactly what it does but this you you more often than not your stuff is wanted by multi-user target because you want it to start like it doesn't need to start before any of the system processes um unless you're building that but for this you totally don't who do we want this service to run under we don't want this running under root we don't like running services under root um and you can do user sami and we're going to say we're going to run under the sami user group is in www.data and this is the group that we want this to be under the permissions so data is a group within unix um this is kind of again isolation before i move on i'm going to answer a question that i saw in a chat that's a great question can we install database and python application in a separate droplet yes you can you can install them in the same droplet you can install them in separate droplets you can use digitalocean's managed database service which is what i'm doing um for a managed postgres the manage postgre doing it through where's my cloud console go here it is i don't know why it's disconnected from everything um i'm using the managed service because if i ever wanted to scale this if i wanted to spin up more i either have to have it installed in a separate droplet and do all that configuration or i can just pay for digilotion's managed service um and then everyone can just connect to it and like it's already dealt with it does backups i'm not a dba um i don't know anything about like database maintenance it's one of the few things that i never really got into in my sre career um i actively avoided managing databases um so there's a managed product you could do it in the same droplet but then it's harder to get let other things connect um you could do it in a separate droplet you could do it as a managed database wherever you want to put it you can put it somewhere so next thing we're going to do is we're going to set environment variables um and this may not be the best way of doing it at the moment because what we're doing is we are going to basically put our data we're going to put our database credentials right here in this so where this is kind of a problem because it sits in a file there's a lot of um you could do this as like like it's going to set an environment variable for it to run now here's the thing these are protected by root okay um a lot of i've i've seen a lot of people like oh we don't want to do that because like it's written in a file and you're right but if someone's already in your server if someone's in your server and they're already root they've already owned you now this does mean that it does mean that they can probably get to the database but they could have extracted that some other way um you could have set up like a vault server and pulled this down um but somehow some like but somehow somewhere there's got to be a file that basically runs this kind of stuff and either like it either some code that runs it or something you could do like an end file this this password and database password is going to exist somewhere um there's probably like much much more secure ways of doing this but in reality in my mind this isn't awful um i might change my mind on that i didn't really have too much time to go and like there's a lot of stuff to make this more secure when i run this in production at my business that makes a million to two million dollars a year no i would definitely spend time to solve this problem if i'm running my personal django site on the side when i run it like this absolutely i don't want to spend that much time worrying about it this is where you have to do the risk versus reward on security is it worth a couple of extra droplets to set up a vault cluster or is it worth all my extra time to secure this or is it there is this good enough for the k for the sake of what we're doing now this is good enough i know people will disagree with me on that and that's fine i'm usually the you'll you you y'all y'all know me y'all watch me do this i'm usually the one that harps about security um but like i said security is a cost versus it's a it's how much work do you want to put in versus how much security do you actually need and you have to do a risk assessment on that the risk assessment for me at this point is that this is okay it's not great but it's not sitting in my home directory like you'd have to be able to find this service which is only accessible by root and if again if they have root access you are you have more you already have problems um techtalk.sammy.cloud is the django allowed hosts and we're actually going to set one more environment and we're going to open this up and check it real quick where's it at here it is i want to see what i named it url shortener settings django secret key environment equals django secret key this is the key that you set um whenever you're just doing like this is this is basically what generates like assault for your csrf token and we're just going to use like a random password generator we're going to say 32 character uh i don't actually know i think 16 is enough i don't know how long does it have to be we'll do 32. i like 32 it's good number 32 characters we're going to remove symbols we'll do uppercase lowercase generate password there we go copy and we'll do this here and we'll close it so these are the environment variables that are going to be passed to it the reason i'm doing this is because i ran into a small bug if this doesn't work i have to add this preload back in but i'm still figuring things out um and that's a thing for another day uh so with django secret key where is our code located it's located at home sammy url shortener api that's where we've actually cloned our code too so that's where it's at how do we want to start it go into that url sami find the g unicorn that was installed in the virtual environment and run it and this is like just g unicorn config and that's all it is this service is pretty much ready to go so let's go ahead and start the socket that was my salt sudo system ctl start gunicorn dot socket sudo system cplc innate we're going to enable it so that way it turns on in case we reboot the server and let's do a status on it real quick and let's make sure everything is good okay socket is running and it is listening so let's actually now let's test all this let's do sudo system ctl status g unicorn okay it's dead nothing's happened yet so what happens is this socket is basically always sitting here listening whenever something activates it the g unicorn service will turn on and serve it so let's go ahead and curl unix socket slash run slash g-unicorn dot sock and let's say loca it's on uh it's on localhost right yeah okay this is good it's a 404 because there's not an index so if we do a status on g unicorn now we see the g the g unicorn service came up and is running so we're good now so like g unicorn is running it's running locally but we haven't done anything with it yet we haven't configured it to where it's externally available so let's go ahead and do that with nginx and i have to speed up a little bit because i'm running on behind on time sudo vim etsy nginx sites available and we can call it whatever we want we're going to call it url short 10r and we're just going to do a pretty straightforward enginex config so this is called techtalk.sami.cloud we got to make sure you get the name server name right techtalk.sami.cloud uh location slash favicon access log off log not found pretty standard we have to say where are our static files where is slash static this is how django um serves its static file remember the django and unicorn don't serve static files that's up to the web server to do um so yes you could do this and do a location static uh root home url shortener api and do that uh yeah and then basically we say location at root include proxy pass and proxy pass to our g unicorn socket so tobias says that my postgres server does not have an ip white list everyone can access it yes you should do this and i should have done this last night but i forgot um always ipwhitelist your database the database is going to be deleted in about three minutes um whenever i'm done with this talk so again i didn't take it but you know what let's do it why not there's no reason not to show you how to do it we can take a break detour taking a detour this is what happens when you distract me so let's go to our tech talk postgres secure this cluster search for sources tech talk droplet allow source these inbound only there you go fixed it's how easy it was to fix i didn't even it took me less than 30 seconds so now only um yeah only our cur i can't talk today only our droplet will be able to access this so even if you did see my password you can't do anything with it so now you're locked out hopefully no one was mean to me during my tech talk that would be mean i wouldn't do that i wouldn't do that to you so don't do it to me there we go i am gonna yeah and i think we're good the django site the unicorn service would have failed had that not connected to the postgres database so it didn't change the stuff there so we're good on that i think we'll find out um then let's do a pseudo oh we have to activate it so to activate our service we say sudo we create a soft link uh nginx sites available url shortener to etsy nginx sites enabled this allows us to basically so what we do is we we put all of our configs and sites and available but nginx doesn't read that whenever we put it in sites enable that's how we turn them off and on it's a really quick little way to just turn off and on sites by just deleting the sim link and i really like that it's a really nice little like way of doing things nginx dash d so that's a test for nginx our nginx is fine so let's restart it and let's go check our browser and see if stuff is up okay talk.sammy.cloud 404 would do that's good better than nothing because there's not anything at the front page remember and then let's say docs okay that's those docs we have another docs docslash gives us the api everything is running but it is not secure so the last thing we need to do is let's do certbot double dash engine x busudo certbot double dash nginx i have i've ran through this like three times and every time i forgot to put sudo there i didn't even put it in my notes always put it in your notes people my email address is mason ideo.co no i will agree to terms of service no i do not want to get emails i get enough emails it is which would i like to activate for we'll just say one the tech talk let me wait on it we're going to redirect https because it's 2021 and you should always redirect to 2021. and if we hit refresh here we are now secure so if i come over here now and i say admin and we say my username was an edgar and my password this and it works good so now what we would do is we would go to url redirects let's go ahead and add one we're going to say test so boring is redirects to my personal website we're going to click save so if i go to tech talk dot semi dot cloud slash test it redirects to mason.dev and you're good now you've created this is a bitly a url api and that is how you deploy to a droplet someone has asked how do you configure nginx with g unicorn on a different droplet you would configure it the exact same way all if you had multiple droplets where this was like maybe you had three and they were running behind a load balancer you would configure them the exact same way because they all connect to the same database they're fine like they're all going to keep the database up and running and then they would just be three workers doing the exact same thing configured the exact same way the load balancer would say reroute them and um you would just like it would route them to it and they would do their work and that's how you would scale uh like that so yes if you would do this on a different droplet you're not going to like g unicorn from one droplet and then proxy to this droplet that's kind of unnecessary you just have three droplets set up the same way with a load bouncer it would do it and they would all do the same work but because they connect to the same source of truth for the database then it's all fine and just works okay we got through the droplet one and i will say the droplet one is the longest and most complex as you can see systemd files there's a lot to that good thing is is the next two are relatively straightforward and quick which i like so we've deployed now to a droplet let's build a docker container um let's go ahead and build a container for our website redirect remember this is a very simple um tool that i built if i can find where i put it here it is uh yes so this website redirect uh basically again just redirects a single thing it looks for an environment variable called redirect 2 and it redirects it that's all it does it's relatively straightforward we're going to clone it get clone we're going to do it here okay so now let's go into website redirect and let's build us a docker file so all we really have to do is build a docker file and then we would just build the docker so um if you're unaware of docker docker allows you to build little containers that are deployable so what we could have done on our droplet is we could have set up a docker ecosystem and just built them as a docker container ship them somewhere and then we can get them anywhere it's a really nice eloquent way of like creating what are known as microservices where you only really do what you need to do um so first we're gonna do is from python there's a lot more to docker than just that that's the docker in one minute um so here we go awesome thanks have bye bruno god you can make it uh we're going to say from python we're going to set our workdir to var dub dub basically this is where we are going to be doing all of our work this is where whenever we run a command that's the context of where it's going to be at so we're going to store our files it's just going to work out nicer um and i don't do a trailing slash here i don't think it matters that much uh before we move on someone asks do you need an nginx server on each on each droplet or the load balancer directly connects to g unicorn i would use nginx for it because you need nginx to serve the static files g unicorn does not serve static files so you would need the nginx on each droplet and then the thing comes through there may be a way to just do it where load balancer to gunicorn i'm unaware of it if i was setting this up today i would be putting nginx on all of them or apache or any web server but you still need like the python application still needs web someone to be a web server someone to be the wsgi and the python application um when we deploy to app platform you're going to find that app platform acts the role of the web server and we don't need that so all we need is the the g unicorn whiz key and the other thing um how do i run more than one site on the same server multiple virtual hosts and nginx um just google nginx virtual hosts um i can't go over that today but i might do a tutorial on it's not a bad idea because there is a trick to running you you can you can fall in some holes if you try to do multiple django sites on an nginx server uh you got to be aware of it whew okay i'm talking fast eggs we still got a lot to cover and we're almost 20 minutes left hope i'm doing good take a drink hydrate you need water you always need water so env redirect to we're going to say https colonmason.dev we're going back to my blog site so if you go to whatever we do this droplet as however it gets deployed you go to it's going to redirect you here we're going to copy app dot py to slash var www slash app dot py we're going to copy requirements dot txt to slash var www requirements dot we're going to again do one more copy we're going to send our g unicorn config which we haven't made yet we're going to make that in a second uh dot p y slash var w g unicorn config.py we're going to run pip install dash r slash var www.requirements.txt so this is going to add all of our code install it now you may be asking why aren't you using a virtual environment this is a fight that people get into um and i stand on the side where i don't think in a docker container you need a virtual environment like people like oh but it's system python you shouldn't be running system level stuff inside of a docker container okay this python's probably going to pull the alpine linux python um and there's almost no system there so creating a virtual environment is just creating more unnecessary overhead in my mind okay there are people that completely disagree with me i get into arguments with them on twitter all the time if you want to create a virtual environment for inside of a docker container hey go for it it's not going to hurt you i personally think it's another layer of misdirection it adds a whole bunch of stuff that isn't needed like it adds a whole another python install that makes your docker container bigger that makes means when you're going to deploy it the download is taking more time big docker containers are bad like they're micro services not macro services don't make it too big i'm a little bit passionate about this topic um i run my python in docker on system docker and i will i will happily stand by that stance if anyone wants to challenge me on it um but for me i'm doing this if you want to create a virtual environment it's not going to hurt anything i mean though this is what i do so then we do g unicorn config g-unicorn config.py worker dash tmp-dir slash dev slash shm app colon app so let's talk about this g unicorn okay we're used to seeing this command this command is what invokes unicorn config we're we haven't written this yet but we're going to set some config settings any config setting we're going to set like two really simple ones but if you want to set some complex stuff and you want it to use a config hey tell it to use this config worker temp dir dev shim this is a weird bug that persists in g unicorn and docker where it doesn't you don't have permissions to access the file system in a certain place where g unicorn wants to so you have to say hey create this local temporary directory and run there um this is an app platform workaround and it's not just not platform it's pretty much any time you try to run it in like in a path type setting so you can chalk this up is to just put it here um this basically just creates a temporary directory for it to run um and then app app are this basically what the app app does let's look at it real quick so app is the name of the file app.py colon app so inside of the app.py there is an app this is saying hey run this g unicorn service on this app if i named this fluffy kittens it would be app colon fluffy kittens if my file was named main dot py it would be main colon fluffy kittens or main colon app so the the way that app app works is it's name a file colon name of flask object in the file that's the secret sauce to it so the next thing we're going to do is we're going to build our g unicorn underscore config.py and we're just going to say bind 0.0.0.0 colon 8080 like normal and we're going to say workers equals two so we're basically going to have two wsgi processes um and now we're good now all we have to do is do docker build dash t tech talk dot this is going to download everything this is going to build us a docker image of our docker container and then now if we wanted to test it we could say let me clear and bring it back up to the top docker run dash p 8080 colon 8080 and then let's say uh i called this tech talk didn't i let's run it buying for port is already allocated who has allocated my port for eight thousand oh it is upset with me there's not a unicorn in my requirements file i messed up okay so let's go ahead and do a python-mvn vn so i don't have g unicorn in my requirements file i made a booboo um is the config executed in startup ie could i dynamically create it you could if you wanted to you don't have to do it the way i did you could um you could probably pass it in as params if you wanted to so we're to do source vnf bin activate we're going to do pip install dash r requirements this is just to install everything we already have then we're going to do pip install g unicorn okay so we've already installed it and then pip freeze require mints.txt so basically what we did is we installed our requirements we installed to make sure we had everything we installed g unicorn and then we added it to where now it's in our requirements file we do the docker build again where we're just going to rebuild the entire container which is just going to copy it um and then we're going to try my docker run again hey that looks like it's working so if i go to local host colon 8000 it redirects to mason.dev because my docker container is running here doing the redirects and it doesn't log anything so i didn't have to log anything i should have had it logged something uh but as you see the localhost port 8000 recruits and now there's something weird here where my um wsl doesn't take control c against my docker image so now i it's been doing it does this on occasion so i can't kill it so now i have to kill dash nine three zero nine zero seven and that kills it how are you getting that syntax highlighting in the terminal this is zsh so this is the zsh terminal inside the windows command line so zsh does that kind of syntax highlighting in the docker builds i did notice that's different maybe it's a new version of docker because i did recently i'm constantly upgrading my docker now because docker desktop for windows makes me get the newest one um so it does that and it's pretty cool um so we have our docker image let's go ahead now and upload it to digi lotion container registry where is the thingamabob there's the thing on the bob i always lose it so the digilotion container registry is an option that we have here this is the one i deleted yesterday let's just delete it that's not what i want to do okay i'm messing up i'm getting off track don't let me get off track i have a digital ocean container registry that i can upload using the digitalocean command line tool so what i first think i need to do is i need to create a api token which i'm going to do over here out of the site of prying eyes and once it's done generating i'm going to copy it okay and then we go back up here to this okay so i have the api token and i do a so digitalocean has a command line tool called doctl let me pull it up real quick so doctl um is a d the digit command line tool that you can use it's a pterodactyl it's hilarious that you can use to just do almost anything you do in the command in the terminal or the console you can use the digital command line tool to use container registry you have to have doctl because you have to do a login with it um doctl off init and then i just paste my token hit enter validates who i am who i say i am voctl registry login we got that and then the next thing we have to do is we have to tag our image with our container registry uh name so my container registry name is registry.digilation.comm edgar uh so we say docker tag uh tech talk registry.digitalocean.commagger slash tech talk we've got a tag we've tagged it now then we say docker push registry.digitalocean.com slash m edgar slash tech talks and now it's pushing our docker image to our container industry this is a private container registry um that you now have that you have access to and now my docker container is here and i can get the sha the compressed size it's still pretty big 328 megabytes is a pretty big docker image um that's python for you doc python images are usually pretty big uh my site was by the way this is an html site and it's 51 megs um and then this redirect site hey it's roughly the same exact same container so it's the exact same thing pretty much um but yeah so okay someone asked say hi for sri lankans hello everyone from sri lanka happy to always give shout outs to people um so now we've uploaded our container to our container registry now if you wanted you could do this to kubernetes you could use docker compose and download this this is how you create the python docker container we're going to move on now and we're going to deploy to a different well we're going to deploy two apps to app platform i have 12 minutes i can do it i promise if i go over the magic voice in the sky will scold me terribly in a british accent which is the worst type of scolding um and i sometimes i say things just to make uh this person laugh behind screen because i can see them so uh we're gonna deploy from github so let's talk about our next one we're gonna talk go through so if you watch my fast api um tech talk we built this uh one-time password api if you didn't watch it go watch it it's pretty dope um and this is basically just an app that uses some crypto utilities that i wrote to create a read write a secret with an expiration everything that connects to redis so um what we need to do is we're going to just deploy this to github so this is digitalocean's app platform so if you don't want to do the droplet because let's be honest that was a lot of work okay i find joy in that because i like systems and stuff i like being in there but a lot of people don't so if you don't if you're like oh that looks awful you're probably right um and then the docker container requires you to have like a kubernetes cluster or something um yeah but you could just upload your code to github just like this so if you remember from my like it's got app.py it's got requirements.txt and crypto you in like my it's just a python app let's go ahead and click next and then what we're going to do here is we have to do a one more thing or we're going to what you have to do some environment variables so the environment variables we need basically connect to allow us to connect to redis so first but first we have to generate a salt i always forget the salt um so let's say salt this is just a cryptographic thing that is used for generating um a little bit of entropy inside of cryptographic algorithms um and this little encrypt button allows me to make it where once it's done you're not going to be able to see it anymore so we're going to have the salt there because i always forget it and then let's we have to connect this to a redis database so we have a redis database let's open up another one right here um yeah tech talk redis and let's go look at our stuff so we need the host so db host is one of we're gonna copy and paste it from here copy db host is this long string right there oh we're also going to say redis ssl we're gonna set this to true we're gonna say yes we do want to use uh ssl for redis the db port is on this server two five zero six one um and then the db password is the last thing i need so let's show that let's copy that again i'm going to delete these servers as soon as i'm done with them so don't try to be mean to me db port or password we're going to encrypt that so now we have all the environment variables that we need and this is and remember fast api is a a asgi framework it's not a wsgi framework anymore so you can't use g unicorn anymore we have a different one called asg or uvicorn uv uv corn so instead of this run command which is the default one that it wants to run we say uvicorn app colon app same thing as with the flask one but we do have to say host is 0.0.0.0 and our port is port 8080 and we hit next we determine where we want to put it at tech talk name new york click next we're going to start off with a ten dollar one one gig of ram one v cpu we're going to click launch basic app and now we basically wait for this app to be built it basically builds its own docker container uploads it and deploys it uh nanashi asks um uh does this you app use build packs such as paquetto i don't know necessarily what what pochetto is off the top map but this does use cloud native build packs so it does detect your code and uses cloud native build packs to deploy your code um so one other thing we're gonna do so these these builds especially on the first build when it's building all the container this can take a little bit of time and i don't want to sit here in dead silence so what we're going to do is while we're waiting on this one to finish building let's deploy our digilotion container because we have see wait you can deploy from github from get lab you can pick a docker image from docker hub or if you've uploaded something to digitalocean container registry you can just deploy it from the container registry and it actually deploys really fast um because you've already done all the docker build stuff it's already uploaded container registry so if you want a little bit of speed on it um and you want to like control your own build process you can build and upload you totally don't have to our build packs will take care of it but i personally like doing this it just feels a little bit nicer it lets me test it like locally docker i like i like docker so i do it now if you remember correctly from our code yes uh yes from our docker file we set this to redirect to mason.dev so that's the that but what we can do is we can actually overwrite this in very environment variable through digilotions.platform so if you were just to run this container on default and never change anything it would always go to my personal site um if however if we come here we just say environment variable we say redirect to we'll say let's just redirect this to digitalocean site and it will be good and it will override the redirect it's running on port 8080 that's the default port you can change it here if you wanted to you totally don't need to um i just run everything on 8080 because it's nicer the run command doesn't matter because it's a docker container it's a web service let's click next let's do tech talk we're going to do it in new york we're good we're going to do five dollar a month drop if you want a little bit less ram it's only it's like i don't this doesn't get a lot of reader x and now we deploy it so let's go back and look at our apps before we do that i think i had a quick there was a question what operating system are using i am using windows and the w the windows subsystem for linux using the windows command line if you want to learn how i do all this because i every time i get asked about like what operating system i'm using how are you doing this on windows my next tech talk which i believe is in mid-june i'm getting the nod from the person in the back of the room i will be doing i will be finally revealing the curtains and talking about how i um how i do this we're going to go through my entire setup we're going to do build it all from scratch i'm going to show you how to develop using the windows subsystem for linux and docker on digitalocean all right sorry not just on windows not even on like you know it's going to be my least digital option talk but it's going to talk about all developer environments uh can you implement cd with this push code build new image do auto deploy so app platform gives you the option to auto deploy on bill on push so if you if you if you push to your main branch and you have that checked it will automatically deploy a new your new service if your service doesn't come up it fails our system health checks it automatically goes back and and does it and like rolls back to the one that was working so uh yes you don't really even have to implement cd you could probably implement it on your side and then i would say just like you know that's what i would do is i would implement everything as a different branch and like once cd passes that's when the merge happens from branch one to the main branch and once you've merged into that then it'll automatically deploy so yes it automatic already does that oh i'm using wind linux on windows okay no no sad tux noises you know if you want my first blog post on that mason.dev site is called yearofthelinuxdesktop.exe um and it's basically how i did it i'm actually rewriting that tutorial for digitalocean because that's an old tutorial okay let's check and see if everything worked so the tech talk one typically deploys really fast and it did you get a default uh application here so if i click on this it is redirects to digitalocean.com because that's all it's going to do like it's just a redirect so we deployed our docker container as an app platform thing and it's just working if i ever want to change this like all i literally have to do is i come here to uh tech talk and i click on environment variables let's send this to um i don't know let's send this to our community tutorials page i literally just come here click save it's going to redeploy and then that's going to redirect there and it shouldn't take that long let's go check our other tech talk sorry our other deployment see if it's done still going let's see what the logs are at okay it's adding the layer for the the heroku uh it's doing the docker the docker image upload on the first one this takes a little bit of time um because it's doing all of the stuff that all the stuff that we've done previously it's done so we basically just have to wait for that to finish um and every day these are getting faster so like we have an amazing team of engineers who work here who are constantly working on this um if you are interested in working at digitalocean we are hiring you should totally check out do.co jobs um you could wind up working on any of these fun tools so digilotion is actively hiring a lot of people right now check out our job board let him know i sent you i don't think it'll do anything but oh it looks like it didn't update properly maybe i have to read it i might have to do a hard refresh on it did i not do it right that's interesting i probably messed something up somewhere annoying me it also might have cached this that could be what if i if i do it in another browser will it work like let's do a incognito window over here and let's see if it works might have just cached the response yeah it catches the response that could be an irritating thing because your browser catches the response um again i would recommend changing this all the time i literally have this here discord.mason.dev gets you into my discord server no rebuilding the app does not make the app go down it does use i believe it uses a blue green deployment where your app will not be replaced by the new built one until it has passed the system health checks now that really is just checking http are you up now if there's something on your side that failed we don't really have any way of testing that like but if the app crashes and like doesn't we detect errors it will not go forward so it does do a blue green deploy for that um let's go ahead and go over back to my other apps let's see if this one's made any progress move still adding that caching layer hopefully i didn't make any errors because sometimes it does that like it doesn't update and i made a booboo um but yeah so this will eventually deploy and it will be fine it'll work just like the other one did so like oh nope we we made we made some we made some progress here so i'm going to hang around for a little bit um go ahead if you have any questions that i haven't answered see as i'm as i'm trying to get ready um come on you can do it okay uvicorn has started and deploy successful get a 404 got a 404 great so now let's test this so again this i don't i write apis i don't put anything on index so final thing http post and i have all of this here and i'm just going to like delete some of it uh what did i call this tech what is this one called this gave me we're just gonna copy that honestly so we do that delete this boom do i make a booboo you can't do https https there we go so now this our api has given us back an id for a one-time secret our message was hello there the passphrase was loki if i do an http that same one again i copy this and paste that right yep passphrase was excuse me was it lowercase lucky lowercase loki now that i mess it up i made a booboo somewhere http that's what that was let's copy this maybe i missed a character somewhere that was the id slash secret slash that and it's not a post got it there we go i must have made a mistake on my thing um and yeah that's the app working in digitalocean's app platform and we got done right on time so nice uh the database is open to connections uh currently yes again i didn't do didn't secure it we will um that that's that's yeah that's fine it's coming it'll be taken care of um how the how the log retainment of digital apps when you deploy the app do the old apps still visible or not or can you pass logs to a syslog server um i don't know if you can pass logs to a syslog server yet um and i actually don't know about how long the retainment is uh we will ask those questions and i will when we send out our follow-up email if you didn't register for this tech talk go to the tech talk page and register because a lot of the questions that i either miss or don't answer we go back through here or i don't know i go i get the answers i give them to the person and we email you out so i don't know the exact answers to about the log retainment and stuff um off the top of my head but i will ask and i will find out and we will send an email with these questions and we will um respond with those so that's how you deploy python applications we've deployed to a droplet we created a container and deployed the container to app platform we could have can deploy it to kubernetes if we wanted to um or any like docker compose this docker swarm even still exists mesos nomad anything that takes docker and then we deployed just an app platform app from our um from just a github repository and just deployed it that's all i have if there's any more questions i'll stick around for another handful of minutes if anyone has any questions that they would like to ask me um if not thank you for coming my next tech talk will be in a few weeks it's up in the chat if you missed it on i will be talking about the windows subsystem for linux i like how i do all of my coding on windows i've been programming for the last two years on exclusively windows i mean using linux obviously because it's linux on windows um and as you saw earlier i had to kill the docker container because the control c wasn't working it's kind of irritating but it's not not a deal breaker especially because if i get bored i can play a game of overwatch and then come back to work and i don't have to go change things um thank you sam uh yeah so goodbye everyone any more questions happy to answer i'll stick around for another minute so we it's that i'm we've been doing this for an hour and five minutes now i will stay around for the hour and six minutes and answer and if more questions come in i will keep answering but i will only stick here in odd silence for a minute if nobody has any questions i wish i had like a monologue or something you could ask my co-workers i can do this for hours like there's a great simpsons clip and it's stories that don't go anywhere it's one of my favorite clips and it's an art form um that i think i learned from just sitting around campfires with my uncles where they just talk about stories that don't go anywhere um and yeah yes this video oh sorry up this video will be uploaded immediately to youtube after so if you missed any of this you'll be able to see it i've gotta remember start saying this um uh how do you on how did dio understand how to deploy from github uh build packs so you you you use oauth authentication to authenticate to github to give it access to your stuff and then it brings them in and then uses uh cloud native build packs um but yes samantha wants a joke i can tell one of my old cloud chats jokes because i don't uh i don't want to spoil the new cloud chats jokes but let me let me go back to one that maybe some people haven't heard one of my what's my good one um i still think the best one was the funniest one the first one first one and it was okay so we're gonna end with this i'm gonna end with a joke and then we're gonna leave the joke today is why did the guy get fired from the keyboard factory the answer is because he was putting in too many shifts think about it thanks everyone for attending i will see you next time to talk about the windows subsystem for linux have a great day and have a good weekend

Channel: DigitalOcean

Views: 20,880

Keywords:

Id: ihaiZWBP4jE

Length: 66min 43sec (4003 seconds)

Published: Wed May 26 2021