State of Linux Container Technologies

[Music] where all the actions [Music] [Music] [Music] okay's people continue to wander in since I only have 35 minutes under the 50 minute talk or an hour-long socking I have to get started my name is Dan Walsh I lead the container to container team at Red Hat today we're gonna be talking about new container technologies that we've been developing over the last year year and a half way I'd like to stop this talk first of all everybody who went to see Scott's talk yesterday about container technology a bunch of you so this is sure it is a fairly powerful shirt of Red Hat and a lot of times we talk about containers and the way I like to describe containers is that just simply processes on a Linux system and if you've one way to describe containers is to say that they're processes that are controlled by three things one of them is C groups of resource constraints basically taking a group of processes putting things like memory CPU utilization and trying to control how much they use so they don't affect other groups of processes on the system the second thing to think about when you talk about containers is security constraints so basically I want to make sure that this group of processes doesn't mean to fear with this group of process so there's no escalations things like that the third thing you think about with containers and stuff called namespaces so namespaces gives you that virtualization feel so it's sort of there's a pity namespace as soon as a process chooses to join a pit namespace it loses view of all the other processes on the system similar Mountain namespaces enjoy the mountain names place everything you mouth and then on is not seen by your parent so your mount table starts to diverge from your your parents mount table is so those are the three things that basically make up containers so see groups some kind of security constraints and then namespaces if you boot up a modern Linux system Arel like a rel 7 system or Noah Noah you would see that pit one inside of a system D that boots up the system if you went and looked at it you would see that you could cat out proc one /c groups and you would see that pig one is inside of C groups that see groups associated with it if you cat it out if you went to proc one and and you would see that you know system DS running with SC Linux constraints it has used users associated with avocados proc one slash status you'd see capabilities associated with it lastly if you went to proc one / NS you would see the namespaces associated with pig 1 so when you boot up a Linux system everything in the Linux system is in a C group has security constraints and has namespaces so by the definition of that because those things being required for a container everything on a Linux system is a container and that's why the shirt says Linux is containers and the backs that says containers of Linux so really the whole you Linux system is built to build these containers now container runtimes are all about basically modifying those constraints so further locking down what a process is able to do on the system so lastly when people ask me can I do that in can I run this in a container I always say can you run it on Linux but the answer is yes that you can run in a container okay so we're gonna talk of next generation but let's start by doing this please read it out loud all text and read this is excellent [Music] [Applause] [Music] excellent okay so since we're trying to do this talk without using the doctor word we have to put off the swear jack for those that I may be native to the u.s. swear jar is what and American households when you're a child would say us where they'd have to put a quarter or some amount of money into a switch so if I say the D word during this I will have to put money in there but the real point of this is to point out that so that the the D word is sort of dominated the conversation and it's really just one form of doing containers and I believe in a lot of ways because of that we've been sort of hindered right five years ago and I'll say it now darker came along and they you know they sort of revolution the godfathers container stuff didn't to take off and all of a sudden it became the only way of doing it and just just by the containers are nothing more than the process on a Linux system and because of that we've had some hindrances in my opinion and what I want to look at is new tools to be able to do this container technology and be able to expand it so when I look at it what do you need to do to run a container so what what does it mean that I want to run a container on a system break it down what I'm trying to do here is break it down it was this core components so when I want to run a container on a system first of all I have to identify what a container is okay and that's really sort of you know what is a container where's a container image so most people when they refer to are actually referring to container images right I want to pull something from doctor Rio that's the site I want to pay for that one so you know they want to pull some kind of application down and and what happened the real side of the dr. revolution was that they standardized on this concept of an image when an image was as a table all in some JSON files so what you do is it created what's called a root efest a root offenses it's a directory that looks like root on a Linux system and then I create a JSON file that basically describes what's in the root efest then I tie the thing up together so I use the you know tired tape archive tool and Linux on my tile goes up now I can have what's called layered images which is basically I'm going to install something on top of that root FS so I tie up the first one and I install something new now I tie up the difference from the original code to the new one in the top wall and I created another JSON file that modifies the original JSON file and I tear that up and that's a layered image ok so there's just nothing more than to power balls and JSON files and you know the next thing you do is you take these tabs so you put them out on a website and in this case we call that website a container registry so a container registry and then we build a protocol to pull those images back and forth so when we came out with these these tie balls are originally there was no standard right there's no standard for it and everybody was just using the de-facto standard basically what doctor did in the beginning and so what they did in the beginning everybody was fine with that for a little while and then all sunflora West came along in core OS had a different technology they had a technology called rocket and what they wanted to do with rocket is they wanted to be able to support out their own application container images and and what they decided to do is they came out with they wanted to standardize on it so they didn't want a one company to be able to control what it is and if you think about the problems of controlling what the data the data images just think of Microsoft so Microsoft came out with you know dot docx format back in the 1990s and what microsoft would do is every single release of their operating system they would basically change the doc format so all of sudden people couldn't send documents around unless you bought the latest Windows or the latest office products right so if you have Windows 95 and also Windows 2000 comes out all sudden people would build documents on Windows 2000 and you wouldn't be able to view on one under Windows 95 and of course Microsoft also was able to get like Libre Office and OpenOffice and all these other tools weren't able to interoperate so what we wanted to do is get a standard org is a standard application core OS that we have to have a standard on what this image format was and so they came out with the app see SPAC now app see spec was different than what was the darker image so there was a problem with that had prepaid for next one so that so all of a sudden the big industry companies like Red Hat and Microsoft and and Google and IBM basically said and said this is gonna be bad what's gonna happen here is also there's going to be multiple different specifications so if you want to build applications they're gonna ship in the future you can have to have an app C version you're gonna have to have a darker version that was my second one and so we really want didn't want everybody having the ship different type of container images so everybody got together and said we're gonna form a standard and that was OCI so OCI stands for open container initiative it was a standard spotty originated by Red Hat Dhaka ranked don't have to pay for that's the company and Microsoft's IBM Google and maybe our core OS and Red Hat and maybe a couple others but anyways they got together and they said it and as of last December that came out with the OCI image bundle format this basically defined what goes in an image so a lot of times we talk about images for now on and say is calling them the D image call it a no CI image it's a standardized image forints based on the original D image but everybody agreed to do that so kora let's actually trigger this long before they were acquired by Red Hat so the next thing you need to do oh the segue next thing you need to do is basically pull down on an image so this is a one of the tools I'm introducing today it's been around for a couple of years kind of weird I'm introducing now it's called scope yo how many people have played with scope yo okay good scope EO was introduced a few years ago and the whole idea is originally what we wanted to do is basically go out to a registry container registry and look at that JSON file associated with the registry with the image and if you think about some of these images I've seen JBoss images they're like hundreds of megabytes I know besides getting up in the air gigabytes size of these images so the only way right now with the to look at one of these JSON files associated with an images actually pull the image so do you want to pull a couple hundred megabytes just to look at them to add at this JSON file that describes the image and basically say oh that's the wrong area it's not gonna throw it away so what we wanted was basically to be able to do a D inspect remote def - remote we did a pull request to upstream and they said no we don't want to clutter up the CLI what we want to do is they said we don't want us to do that but he said it's just simple it's just a web service just two web protocols and you can pull down the JSON file build your own tool to do it so we built the tool for that called scope yo so scope EO which means in in Greek remote viewing was a tool to basically look at a remote site and just pull down that JSON file associated with a container image so the guy that did this on my team Antonio medaka actually decided to go further so originally just did you know inspecting images to pull down that but then he started to say well I can fill the entire container image protocol right the ability to pull these images back and forth between registries he basically built scope EO into a tool that could move images around now scope has become really cool because it can actually transition from different formats so you can actually copy down an old CI format store it inside of a inside of [Music] inside of a darker demon you can actually pull it to local for local directories you can translate from the original image formats the new image format but the really cool thing is you can actually move images from one container storage to another or one container registry to another so a lot of people now are using scope EO to actually move images around their environment and we're getting a lot of uptake in this so we were working with Clara West to try to get Clara West to embed scope EO into rocket and they said they don't want to be vetted tool CLI tool ins who rocket what they wanted to do is basically just lose the library the scope it was using so that library became containers image so github containers image is now a library for moving these OCI images and old-fashioned images back and forth around the environment you move between registries and you don't need to have any route based tools so you can basically sit there as users they copy from say my internet based container registry and copy into my internal container registry or copy the files locally so we became a mechanism for moving that image from the registry to the host the next thing we needed to do is basically take that read that image and basically explode it on disk right in order to run an application and container we have to have that route FS re-established so we take down those one or more layers and reassemble them the way you do that in Linux is what with things called copy-on-write file systems okay you might have heard of overlay device mapper butter FS is a whole bunch of them so we basically took a lot of the tooling that we had worked with with the upstream and built it into a little tiny library called container storage those ability to explode images onto a copy on write filesystem and the last thing you need to do when you run a container run around the container is you actually have to basically what does it mean to run the container and luckily OCI is standardized on that so there's stand-in meccans verónica Taner and that was fit also specified last year by the beginning of last year as the OCI runtime specification so inner OSI at run time specification says that I pull down the image and that the image had that JSON file that tells me how to run the container well I also have input from the user and I might have input from whatever tool is putting this all together and I basically want to take those three inputs and combine them together so user might come in and say I want to run in a privileged mode or I want to run without this capability or I want our volume mount in this stuff so we need to take the user input the application that's setting it all up of the container or I'm gonna call it container engine and then the last step is actually to take the stuff from the image and am undissolved that together and basically writes out on another JSON file so that JSON file becomes it becomes the OCI configuration and it's part of the runtime the OCI runtime spec defines what's in that JSON file as well as what's in the root of s so it says put a root of s on the system you put this JSON file between it and now I launched an executable that understands the JSON and configures the system darker rink basically gave the first tool to do that called run C so run C was the first implementation or so the de-facto implementation of the OCI runtime specification just about every tool that runs containers now in the universe uses run C to create the container okay so run see so this is the steps that you needed to do to run a container on your box right everybody agree with that anything missing [Music] okay so we don't need a big fat container demon to do all those steps and I have a big pusher against the big fat container demon because the big fat container demon here we are five years into containers and there's only one way to run containers everybody knows that if I ask you how do you pull an image you tell me the deep pull by actually out of pushes you said deep but how do you build it deep built and everything goes through this one air is the problem with the big fat container that is the biggest problem with it is we got least common denominator of security so needing to build a container is much different than needing to run in production I need a lot more privileges to be able to write to the Container image than I do to basically when I want to run it say 'under kubernetes so we want to do is basically take these pieces of pod and reassemble them and redo different types of tools for running containers each one with the least privilege now later on there's gonna be a talk that talks about some of the security features that we've been able to do by breaking apart the big fat container be I work for open shift so everything that I do tends to be either for open source or I am instructed to do it for open ship so when I look at what open ship needs to do to run content OpenShift is red-hats kubernetes you know our enterprise version the Koopa net is really what open shift is plus plus okay we have other features and other things we've added on to kubernetes but basically if you want to get if you come to red head you want to buy kubernetes from us we will sell you open shift so what does open shift in kubernetes need to run a container they need those four first four things but they need CRI so there's a little story here core OS again Koro West came along and they wanted the original version the kubernetes embedded docker all over the place inside of the code core core OS came along and they said we want to support rocket inside of kubernetes so they wrote huge patch sets the cadence basically sent them upstream to kubernetes that basically said if def rocket do it this way else do it the old way and the kubernetes developed the developers at the time of the upstream kubernetes said wait a minute we can't do this because we do this for rocket then all of a sudden gotten some other container engines gonna come along and say we want you to support our container runtime as well so what kubernetes did is they wanted to turn it on its head and they basically said you guys implement a small daemon and we will talk to it and we will talk to that thing called via CI r ISO contain a runtime interface so kuba needs to find an interface that he will talk to container engines with and then if the container engine implements it kubernetes will very happily do that next thing is coupon that needs needs to do when it talks to a container engine is wants to tell the CRI that needs was it's gonna tell the CRI it needs a container image CRI needs to pull the image from the container registry needs to store it on top of a copy-on-write file system and finally needs to execute a no CI runtime anything look familiar from the first part so we have all these tools another one of my members of my teams when this happened basically said you know we could take our standard building block tools here and build our own CI right and that thing was called trial so CRI also the CRI stands for container runtime interface for kubernetes and the O stands for open containers or OC I open container images so we developed a small lightweight team and that basically just implements what's needed for kubernetes to run containers in the environment and we called it cryo so cryo is an OC I based I already said that so scope is totally tied to kubernetes to CRI only supported uses containers for kubernetes nothing more nothing less let me beat this to death cryo loves kubernetes kubernetes is it cryo is a you know she's very loyal to her man she's never gonna go anywhere she might get mesosphere comes in and says you know that's too it around her and stuff like that but she says no friggin way now we got here definitely not not even in the ballpark this no way and definitely not okay cryo is only all she cares about is kubernetes okay just kubernetes so overview of additional components so there are additional things we needed to be able to do cry out and we'll talk a little bit about those so one of the things we need to do is basically translate the input from kubernetes kubernetes has its own specification of what he wants to do to run a container but we have to translate that specification to oshi a runtime specification so there happened to be a tool inside of OCI called OCI runtime tools actually written by one of my guys but basically you can take input from users a library that will take input from users and generate an oci runtime specification so we use that inside of Craiova next thing we need uses this thing again core OS comes along we needed a way to configure networks so networks is kind of a strange part of this whole container world and that we needed networks too you know we want to allow different virtual private network tooling to come along and build and be able to plug into the container environment there's lots and lots of companies building their own sorta either hardware base no software based antenna net working I mean so work or West is defined a standard called CNI was container networking interface to use to allow other people to plug in and so they've been used with flannel we've opened a light open us the end I think the open shift has their own version so lots and lots of people are building content and networking interfaces lastly to run containers we need a way to monitor the container so when I launch a container on the system using no sea air runtime it just goes out and configures the kernel you know those sea groups and security settings and namespaces launches the process and then goes away so at that point there's nobody watching the container there's nobody sitting out there saying to the container exit right wrapping it and then so we needed a tool to be basically watch the container and basically that's called con Mon we wrote it in C because we wanted it to be as lightweight as possible and it basically monitors it takes care of logging what's the output so when you run containers you usually watch what's up going to standard out and standard era I mean handles the tty it's service serving attached clients and it's a textual basically the figures out if the container died and then writes the status to a file so that now any container engine that comes up can actually go to conmen and basically figure out what happened or you know Carmona likes it with the container but it'll record the data that happened so the pod architecture when you're running kubernetes in your environment coop anaise runs pods it doesn't run containers now container pods are basically one or more containers running together and the pod there's also this idea of what's called an infra container our pods container and what happens when you launch a pod under kubernetes is it launches this little tiny contain a program that basically goes to sleep and it just starts up and then it attaches all those namespaces to it you have to have a process in the original name and then it will add containers to it so if you looked at under cryo what happens when you launch a pod we launch the infra container it has one conmen listening to that and then one or more containers get sloshed so basically this is what the whole infrastructure of audience structure around the cryo so right we talked earlier about how much cryo loves kubernetes and the way we're trying to prove that is basically we have the biggest test Suites every test suite we can find we run before anything gets merged in the continuance trial so we don't want to cry out to ever break and no new features ever break kubernetes so right now we're running and I know it's probably much more than five hundred for this nine full test weeks to get a pull request into into cryo at this point is pretty difficult right you have to jump through hoops you have to make sure that everything is possible state no Piazza merged without everything passing cryo came out was fully supported as of last December one dot negative my engine is wanted to call it 1 0 so we released it back in December I hated the fact that we called it 1.0 so the next release we called it 1.9 which works with kubernetes dot 1.9 then we hit released 1.10 which works with kubernetes 1.10 anybody heads of the guess what works with 1.11 yeah okay so 1.11 works for kubernetes 1.11 we are stocking the hell out of kubernetes okay the goal right now I'll talk about that in a minute but based on the goal for openshift 4.0 is that will support cryo by default right now we support both trial and doctor under the covers but the goal is that 4.0 to support trial by default cryo is now supporting red is now running most a lot of openshift online so if you go on openshift online you're using cryo if you go to Microsoft you want to launch a kind of container you're using trial okay so cryo is actually getting out there but a lot of ways I always tell people I want try out to be something you ignore right the real goal here is to make running containers in production boring okay I often ask people they say alright you use this in the background you use this in the back end I asked them what file system do you use I don't know what file system I have on my laptop this is the X t4 is the next FS I don't know and I don't care the only time I care it is when something breaks and so our goal here is to make this thing just blend into the background it's just that it's just a feature underneath coop images so what else does the open chef need to do to run containers after runs you know and use this cool kubernetes well it needs the ability to build images openshift has this concept called source to image where user just check something in to get does a push and all of a sudden the container poops off the back end of the open chef right so we need that container image to come out of the end so we needed a way to support that for open ship and we need the ability to post these things to container registries so this guy right down here is an island I'd like is working with me last year at Def Con check and we're sitting there together and he's in charge of containers image and I always kept on saying to him that I need a tool for building containers I wanted to call utils for building containers I said you know it's just a brute offense I need to create a brute efest I rode up higher ups in JSON file put it together and build it and I said I needed some coffee and write it say you got that you've got containers image could we throw together something to do that I told him that in the morning while we're at DEFCON and by that evening he did a five minute talk showing how he would build container images using container storage and so he said what do you want me to call it I said I don't care what you call it just call it builder what difference does it make and then he came out with this and so we and the last thing here this is not the current image but this image was the first image we put out of it this is a Boston Terrier and supposedly on a hard hat as soon as we tweeted out that we had of it the icon for this people came back and said why do you have a dog with tighty whities on his head so I still live it it's a so it's much more of a hot head nowadays but I'd like to leave it just for that joke okay so in the coloring book that hopefully you guys picked up if you don't come get me afterwards this is a builder is represent represented as as a dog and I think it kind of looks like now on don't you [Music] okay so builder came along and then again it's my my idea was kept core called utils for containers we wanted to have a simple interface for it so you know we needed to be able to pull an image from a container from a container registry to the post and so we built build it from fedora so what this does is it goes out uses that container image to go out to a container registry pulls down the Fedora image off of a container registry to the local system puts it on top of container storage and then creates a Builder container okay containers a way overused word in this world but basically has all the data that's associated with a container in the next step we need to do is we need to model the container right I want a mount point I want that root of Fest Mountain on my sister and I just want to be able to write to that root of Fest so we built build a mount and that basically brings back a mile point okay another segue anybody ever hear with this command anybody know what this command does there's copies content from a container image to the host or IRA copy stuff from the container from the host into a container image really cool really cool I saw that and I said I'm gonna steal that idea so I decided to go off and build my own tool and I called the copy and I put it into coil utilities on the system and I built it yeah it really you know works really well but once I saw that work really well I decided to build another tool so I built the tool called PNF sometimes you call it yeah I used to call it yeah I might call um again in the future but basically with this tool you can actually install content into a container root offense so I just added the - - install root and you can basically install Apache into a empty root of fast and do it but I said that's cool Alden that's another troll I invented a tool called make so with the tool make I can actually do this thing called dester I decided to come up with this concept of dester and I could basically set it up to the point to a root offense so basically what I'm showing here is you can basically use anything on a Linux systems actually populate what's going to go into your container so the next thing you need to do is populate that JSON associated with the container image and we have a tool called builder config and so you can put things like entry points environmental variables all this different stuff that you've basically put into containers image to identify what the container is and then finally we want to take that container image and actually have a container and create an image right great and OCI image on the system and so that's build a commitment and then of course I want to be able to push it somewhere I pushed it to a container registry so we have build a push so with this tooling and by the way all this stuff here no big fat container team right I don't need a demon to do any of this stuff so I can do it not only that I'm showing it's running his route here with rate with the current builder we can do it as an on route we can do all this stuff taking advantage to use a namespace we're able to do this all is non route now try it again every sin check one two three glad you asked so builder also has some support darker file ok docker file has become this sort of the facto standard I like to think of it's a really crappy version of bash but shell script but basically it's become this two facto that everybody wants to support so we actually had the support with builders who are using darker files so we built a command called build builder using builder build using docker file and basically as the same syntax that you would expect for running builds on it about the costs were engineer so we're all lazy so we actually have build a bud so build a bud and no it's you know anheuser-busch is not involved in this decision but basically we can build container images using docker files [Music] well it's not called bilder file but I decided to write this really nice scripting language and I called it bash so after I wrote bash I basically have that you know lots and lots of tools out there to build container images and the whole idea here is that what I really wanted with bilder is to basically provide a library or low-level command-line tools that other people could build higher level content container languages so we want others to build it we're looking at OpenShift is looking to basically replace right now the source to image is actually injecting the darker socket into the containers to run to do bills a lot of times I tell people that that is probably the most insecure thing you can possibly do if you want to give people access to the so that I tell you that just go in and set sudo to known root and turn off your logging because if you give a non root user access to that socket that's what you're doing if I go and do evil things or on the system as via via the doctor socket I can then destroy my container and there's no record of me ever doing anything on your system so never give out that socket to a non privileged user so we want to do with source the images basically stop injecting that lots and lots of people around running container builders inside of us inside of kubernetes and what they're doing is their volume mounting in that socket okay which is equivalent of giving them route on any coasters they're doing it so we want to be able to do builder inside a source to image and stop injecting the socket ansible containers is also looking at potentially using builder to replace and basically using ansible as your sort of instable playbooks for defining what's in the container image so what else does open ship needs to do we needed to ability to diagnose problems we need people to be able to play in this environment so we decided to create this new tool and we called it pod man so quad man is part of the lip cloud effort so we wanted to be able to basically build a pod manager or a container managing tool and we wanted to base with this tool is just a CLI command line tool that can be used for managing container images and we based it on top of what everybody knows which is a darker CLI so pod man is now out we're actually releasing pod man on a weekly basis we've been doing it for probably the last six months just k5 on 8.3 so we release it ages the month and the third week so at the end of the year we're gonna be in trouble so we have to have one zero by the end of the year coming about naming system gone but basically you want to list the containers on the system if you want to run a container on the system if you want to exact into an existing container if you want to list the images out on the container basically we've tried to copy everything in that CLI possible that we care about obviously we're not doing suam with this command but we've had most of the commands are all done and lots and lots of people there was a great tweet that came out back about now I guess it's back in May and I love this tweet he says I completely forgot that two months ago I set up an alias of dr. equals pod man and it has been a dream so he's been running for two months at this point without use with the pod man of course that's several month old one so next question down comes down and says the only downside there's no book I'll talk about that in a second next one's down Joe Thompson replies and says so who remind how did you might figure out that you were running doctor and Padma instead of doctor and he said I executed dr. help and it came out with pod man help I think I owe about three-quarters so so what advise you to do right now is go home try to sell it try l-pod names available on fedora rel sent to us Ubuntu and it's fully supported on Susy open Susi as well so it's basically gone out we have lots and lots of contributors to it you guess what no big fat container demon okay it works like a exactly works sort of exactly what you expect not a client-server operation the pod man is really really cool and does almost everything you can so we talked a lot about containers there's handed out the coloring book before and I think I'm just about to run out of space so we have two other socks this afternoon now Alan's gonna be giving a talk and I'm sure going back and attacking me so I'm gonna give a deep dive into bilder and then kurbashi and Sally O'Malley gonna be talking about all the difference I said there's lots of security stuff that we're able to do by breaking apart containers so they're gonna be cut talking about that later on the stuff to notice to look for those talks you can take the photo of this and the presentations be there I can't how the answer one question I guess yes [Music] is there any tool currently that can update a tag on a remote on the container remote registry any tools actually someone asks for that and the answer is that has to be built into the container protocol container but basically the protocol that talks between the client and the server and Vincents raising his hand back there because he's gonna point out that they're working on a standard now to find that so that what you're gonna tell me been static the the you can drop a coin and the docker registry API not the docker registry code base with the docker registry API has now been donated to the OCI the open containers initiative as the distribution spec it is the API that would enable a feature like that but it's not really up to the client tools right now they would have to do some shenanigans like fetch the image and then retag it and REE push it so that would be the place to look for it osya open containers / distribution spec right so we actually we had a big bug report that someone asked e for that in scope yo but we have to get it into you know we needed to get into Quay and out of factory docker IO and so we really need that to be a standard how you interact with the container registries to be able to do something like that anybody else everybody loves this idea and they're all aliasing it on their machines right now actually all right anybody want to talk to me I'll be around and thanks for coming [Music]

Channel: DevConf

Views: 199

Rating: 5 out of 5

Keywords: DevConf, Containers, Orchestration, Red Hat, cri-o, skopeo, Buildah, Podman, Linux, Linux kernel

Id: c8uc7unZ8po

Length: 39min 16sec (2356 seconds)

Published: Fri Feb 22 2019