Docker as a replacement for virtualenv

uh so I I guess we can get started here um this is uh Docker as a replacement for virtual environment uh but that's just a title uh really I'm here to convince you to not use any of those tools that are up there and to use Docker for your environments in Python um I'm David Felix uh you can find me at David J Felix on Twitter GitHub probably everywhere else um I've been using python for nine years I actually learned it on stage at aai Ohio uh somewhere around campus I don't actually know where it was um and I'm a platform engineer at Kroger digital uh there's a lot of us we're in Cincinnati Ohio or blue as uh lots of fast moving teams um lots of interesting Tech including Docker there's a little bit of python uh and we're hiring so come talk to me um so right now for environment management uh there are a lot of ways to do it um there's a little asterisk at the bottom I haven't used a couple of these and I hope not a whole lot of people have manually managed their environment for an extended period of time past like you know just using pip uh on the system level there's also Pi en EnV um which is different from PI V EnV which is VM in uh 3.3 I actually have never used that I started using Docker before that um virtual M vagrant uh and vagrant has a number of backends uh but I'm here to talk about Docker because last year when I was up here um I was presenting about async IO and 100% of the questions that I got were about my Docker setup so I figured that might be worth talking about so virtual amp why why are people using it in the first place hopefully you're using something to manage your environments like I said manual is sad face um well python likes to put all of its modules uh very close to its interpreter not necessarily next to it but um they all end up in one place unless you manage your environments for that um by using virtual environments you can you can isolate them from both your operating system which likes to have its own version of things um you can also separate them from other applications that you have out there so you can keep you know this version for this application because that's what you tested it with and this version for another hopefully they don't Trail off too much but things happen especially in production um and keeping that that isolation uh really helps you ensure that things are going to run the way that you expect them to every time and that's a big theme that I want to stick with is that if it's not running the way that you expect it to it's it's not immutable it's it's mutable and handling your environments in an immutable manner um is what virtualization and virtual in is all about so um obviously I wouldn't be pitching another solution unless there were some shortcomings or or um deficiencies um virtual en P en Ven and manual Are all uh dependent on the host OS so with um with like OSX you're going to have a slightly different setup it's not going to be very different because python is is fortunately crossplatform but um with cpython you'll end up with um differences in the C modules again hopefully that's not a lot but Library avail ility will will definitely be a factor if you run into something like PBC um where you need the the backing library for it or if you're tying into um something that compiles slightly differently on um on Windows or OSX than it does on Linux um the availability of libraries are is is going to have a big factor on how your modules get brought in um it's a lot of stuff that you don't generally want to think about in Python A lot of people will encourage like you know to to bring in modules that are entirely python because it avoids all this nonsense that we're trying we don't want to deal with um one another big thing is that there's a lack of unified tooling with these so like getting from nothing to having your virtual environment set up um tends to be a little bit of tribal wisdom uh you know you bring in your requirements file and hopefully that has everything but if your system requires something you kind of have to have a bash script or you have to have a CMD and that's all going to be platform specific um but the biggest one that I don't like is that it's mutable so uh what that actually means is if if I were to go into my virtual environment I can change it at any time uh so if if I'm working on my virtual environment and someone else says oh I like this one I'm going to continue using this because that my app uses all the same dependencies and they update it that affects me and that's that's not good behavior but I haven't done anything to stop it so uh what can I really say about it besides don't do that man you know um so I I I'd like to pitch a a way to solve some of those things um if you've used VMS that solves a lot of it uh you get the whole image down um the problem is that they're very heavy I I actually used vagrant for a while uh I really liked vagrant I've shared vagrant files with people uh shared virtual boxes uh virtual boxes through vagrant with people um but it it is really expensive uh I think the image that I was sharing was about 2 gbes um it was an entire Ubuntu image it had everything and it was just running Ubuntu in OSX or running Ubuntu in Windows um it was pretty expensive to orchestrate if you had two of them now you have four gigabytes and it it was pretty colossal um but it kept everything sane uh aside from the massive use of resources um again very good for mutability and vagrant is still pretty nice even with the docker back end so I'd encourage you to look at vran if that sounds interesting but I actually want to introduce Docker which can be used as a back end for it or can be used independently as I'm going to show you here um Docker uses a different type of virtualization called containers uh this has been made famous probably by uh Google's very secretive Borg platform um which they consider to be like their secret weapon for moving their application around on different servers um it's it's similar to chroot or BSD jails uh and how you isolate processes within an operating system um it doesn't use a whole uh operating system virtualized on the processor level it's actually using libraries that are available to the operating system to virtualize access within the operating system um a big thing for me though is I work in a not only python environment it's um we have Java we have uh JavaScript we have python we have closure so having it not just be python is actually a really big benefit um and I think that shows in the community uh which I'll show you in a in a bit um the big thing about containers though is that where I had a 2 gigabyte VM uh now I only have the libraries that are necessary so I can base it off of an auntu image but I only have 600 Megs of essential libraries and that's like a little bit overboard in terms of essential like I have bash and I have uh I have the C library that comes with that version of Ubuntu so the essential portion of that is still rather unessential um but any of the libraries that you need can packed into that container and shipped that way so it's not a tribal wisdom of here's how you install this here's how you turn on the virtual environment it's here's the image and it runs I've already set it up for you um the docker runtime takes care of all that and what's really interesting is you can do that remote or local so a lot of the early workflows when people sit down and try Docker the first time they log in on their own console and they're running ER locally but in reality it's a it's a TCP command line that's just talking to their local machine you could route that to a remote machine and say go get this image run it so that provides a lot of potential for orchestrating these containers in the future um it's not necessarily something you want to think about early on but having this high ceiling means that you have this this full ecosystem and full set of utilities that really help you take a virtual environment from just here's how to make it not break to here's how we run it sta uh and reliably and continue this idea of like our Ops is all in code so um I I tend to think that Docker heavily favors immutability um everything in the way that Docker operates tends to work in a uh forward only motion and um it actually on the back end uses a copy on right file system where uh any changes are actually moving forward so you can verion it backwards uh and work with that so uh I'm gonna pray to the demo Gods here and we're g to try uh to show it off a little bit if I can find my mouse it's on that screen now it's on what screen is it on yeah I know but uh all right there we go okay can everybody read this huge font is that big enough for everybody okay um so I have Docker running locally um and we're gonna we're going to issue a couple commands here so um one of the neat things about Docker is that it's got It's got uh Community contributed um uh Docker file so a Docker file is essentially a a recipe for setting up an environment uh what we're looking at right here is the official repository made by the docker uh employees for Python and you'll see there are a number of different tags so we have every version Under the Sun of 27 uh we have 33 34 35 36 Alpha uh and then we have variants on top of those so the default one is is based off of Debbie and Jesse um it's going to be similar in user experience to like a uh an Ubuntu image um but with slim it's significantly smaller it's instead of 600 megabytes we trim down all the libraries that aren't essential and get to just usable libraries and we end up with about a 100 megabytes um Alpine is even smaller at 18 uh and I there there are a couple other differences um in general you want to start out with a a just normal one and then start trimming it down as space becomes a problem it's a couple megabytes here or there so what I I did is um I actually pulled down a couple of these because I don't trust the network uh I would normally do something like Docker pull Python and then the tag so I don't believe in Python 27 anymore so I would do this and normally would download all of the layers of that image I already have it so I can just um I can just move forward with that so um normally when you're working with this uh you would have a docker file in your uh in your Source I'll show you that in a second but um that Docker file would say how to move your Source into the container and how to run it independent of you uh right now I'm going to show you the inside of the container and and give you an idea of what it actually looks like um so we're inside of a Docker container right now um I'm this is this is a MacBook it's running OSX uh and it thinks it's on Linux um so I can you know print right a that nice um but more importantly I actually get a lot of utilities there so if instead of just running the container with its default action which is python because that's what it's set up for I can actually get a bass shell inside of there and deal with a lot of the the background tasks that you would want in an environment so um if say I need to move files around or I have I have JavaScript that needs to be compile or transpiled and packed um before I start serving it I can do all of that as part of the build process on my container uh because I have utilities like bash available in the container and then I can slim it down pack it out to uh a repository and any Docker runtime can pull it down and run it as I published it so um now I'll I'll show you what um what a Docker file uh looks like so let's look at the oh boy I lost it 3.5 so um here's the the I mean this is the source for a Docker file and it's it's pretty simple but um the way that it works is that it builds in layers so this from command says that this is a new layer on top of something else someone has done so there's another container out there called build pack depths Jesse so this is going to be a Debian Jesse image that has all of the stuff that I need to build my my project um it's going to run appet and clean up all of the Python stuff right uh and then it's going to do everything that if you were to to try to install python from Source you would have to do um and it's going to leave you with a nice little command here that says run Python 3 as a default command uh so when I try to use that that container that's what it's going to try to do and it'll give me the the prompt um but if I wanted to say uh let me get out of here so I'm in the container right now leaving the container back in my pretty prompt right um uh oh come on okay um so let's say I have a flask app and I shoot sorry uh let's so I have a flask app and um that's all I need to run it so uh I have this I have this from python 3.5 slim I don't need the full size one I'm not using a whole lot of operating system utilities uh this will bring me down Python and then like a little bit of of operating system system um convenience like bash it'll bring me down libc it'll bring me down everything that I need to build modules it bring me pip but nothing too heavy like all of Jesse would bring which is just like you know it'll bring over the proc fi file system and stuff like that so I set the working directory to uh to opt because that's where I like to put things it doesn't really matter uh I copy my requirements file over from The Source on my box that I'm building this on um and then I install those requirements so that that would be like if I was manually installing them but I'm not manually installing them I'm installing them into the docker machine or into the docker um container and then I'm providing that container for uh everyone else to run exactly how I've done it and they don't have to compile it they just pull it down already built so any modules that I build here if I have C extensions that I need to bring in GCC for and then remove GCC for all of that gets done and gets turned into just a file that they download um so then I copy uh this this can all be done as one step I could copy both of these at the same time uh I do it this way because the way that Docker works is that it can detect when file system changes are going to affect it and it will rebuild from the step that it needs to um so I copy my source after my dependencies because I feel like my source changes more than my dependencies so that just means that when um my whenever my requirements file uh happens to change I'm actually going to rebuild everything after that um I'm exposing Port 5000 since that's a default flask Port uh and I'll show you how to map that uh but then I'm setting up the the command so that it's Python and run my hello Pi so let's let's take a look at um hello Pi simple flask app nothing too special um but if I wanted to run it uh one of the nice things is that Docker not only Prov provides this file system isolation and process isolation it's actually providing a network isolation too so these processes um by default nothing can get into them there's there's no in andout of of network traffic there's nothing that I don't ask it for so when I ask it for 5,000 when I do Docker run uh I need to pay attention to one of these screens not both of them um I can do this uh sorry it Trails off a little bit there um let me actually do that so uh what I'm what I'm telling it here is this- p is a a port mapping so I want it the Local Host port to map to The Container port and that'll actually occupy the port as if it was running on my machine uh but I'm explicitly telling Docker that traffic is perfectly fine for it um anything on that Port TCP or UDP will go in um but only because I've told it to do that uh one of the nice things that Docker provides is that it can do virtualized networks between containers so say I have uh a nodejs app talking to a python API uh and I want them to explicitly talk to each other but I don't want any funny business with it talking out anywhere else I can explicitly set up that Network and no other containers that people Deploy on that host are going to interfere with that they're directly linked to each other and that network is actually becoming part of our environment so that's something that and just can't do uh not without a significant amount of overhead with IP tables um and a lot of utilities that frankly I I didn't have the time to learn um I think I'm just about out of time here so I'll open it up for questions uh

Channel: Next Day Video

Views: 34,612

Keywords: pyohio, pyohio_2016, DavidJFelix

Id: ETL-_W1W8gY

Length: 19min 17sec (1157 seconds)

Published: Wed Aug 03 2016