Make Stateful Applications Highly Available w/ Docker Swarm Mode (and Docker plugins!)

man I tell you what doing these in 20 minutes there's a definitely gonna be some people that'd be backed up so without taking up too much time glad to see so many people here during their lunch break being able to see this it's awesome crowd because you think in a community theater with everything going on behind here it'd be tough to get a lot of people but this is really cool to see and glad to see so many people so I don't want to take up too much time I don't want to kind of really dive into a demo so I'm gonna talk a little bit about how to make persistent applications available within inside of a docker swarm mode but also utilizing the new manager docker plugins as we kind of solved today's keynote the whole enterprise has kind of got this this kind of push behind it and with that comes a drag of all these applications that we've been using for quite some time and you know the idea is that not everything is going to be greenfield right and not only that is there's data right we have this false sense of of idea that all we want to do is run docker containers as stateless and this was a quote that was made famous at mazes con this past year and you know we don't technically have like a data list status data center we want to figure out a lot of people want to say like oh we'll just go ahead and we'll just have this database live over here and we'll run everything stateless inside our containers but you get a lot of ability to actually have everything inside of a container in itself so even if you want to say like okay I can run the postcard servers somewhere else and we'll stick to our old virtualization backup technologies or whether its replication inside the database itself you actually get a lot of benefits by container izing the database - right I mean if I think about it and that VAR Lib my sequel directory lives off in some sort of storage that means the container itself actually encapsulates all the computing power that makes it super portable at that point right like I don't have to worry about just the virtual machine that owns it and then I got to worry about all the configuration inside the virtual machine everything like that I can just spin up a new container and do that as well and not only that there's a lot more things that go into this and as you can see I had to translate those over to Google slides earlier because my Mac was kind of giving me crap so pardon the the bad formatting but the top 10 at the top sorry the top there's ten of the top 30 applications inside docker hub today actually requires some level of persistence alright that's a pretty staggering number so in my people that are maybe new to this space you figure out like well how do I know which applications are actually requires some level persistence the easiest way to do it is you go to docker hub or you try to find a something that's your building and you look in there and you see this line that says volume and you see a data directory a path right next to it that means there's data that's required by this particular image right now with the servers and engine X in the self like we can we can throw config files at it all day but something like a database or anything like that you know these are these are these are applications that we all use right there's CRM their job applications they are databases all these different things requires some level of persistence to be able to say if I lose this container or I lose this host right I mean we can even utilizing local volumes forever but I think that we've been around long enough to know that local volumes only get you so far right because you lose the server you've basically lost all your data anyway right so you got to figure out well how would can we put this on some sort of centralized piece so I'm gonna talk about Rex rate a little bit anybody heard of Rex right before all right that's some awesome amount of hands so for anybody that hasn't heard of Rex right before we are the leading storage orchestration engine today for docker and for cloud native workloads in itself and I'm gonna go a little bit and talk a little bit why this matters not only to just like operations people in the simplicity aspect but as well as developers and other storage companies that actually want to figure out well how do we get ourselves into this ecosystem so a little bit pieces about what this does and really what Rex rate does here and how it's helping your persistent applications we are do we are providing all of the orchestration for it as well so you think about everything that you have to go through for volume creation detaching attaching removal all these pieces it's actually orchestrating everything behind the scenes a lot of people say oh we're just using blank NFS shares and you know I'm mounting it them all the hosts and then we lose a host so what then I got to go to config file on the share and and remove that host name and IP address and that's so that's a lot of manual intervention right so what we do is we automate all of the orchestration between all the different storage platforms that we do support we do consider ourselves enterprise-ready rex ray I actually package itself with a few different key elements such as a fully built-in and functional CLI you think well okay well obviously a lot of storage players here on the floor like what makes that a leading or a competitive advantage well think about if you have a stale mount or a host that's in a lock state and now you've need something that you need this volume available on a different host well how are you doing that how are you forcefully like making that unlock happen Rex ray is a full-blown CLI that has all the volume operation cycles or volume lifecycle built into it so I can say rock Rex are a volume detach - F and force this thing off and kill it off the host and make it go somewhere else anybody else that you might talk to to say all you got to go to the GUI and you got a you know type in a few commands or whatever it is to be able to do that at the same time we also have built in a high availability function called preemption and what this is doing is you have will say we have one container running on one host it has a lock on a readwrite volume and now I will say this host fails or would you say we spin up another container and now this container is ackers is requesting access to that volume well what its gonna do it's asking to talk to the storage back-end forcefully disconnect it from the previous host and attach it to the new host right so now we get high availability built into our orchestrators which I'll show you as well this is also completely open source it's all on github and Cruz you go to Rex rate code LMC calm that's a splash page for it gives you blame or information but then also has a link to the github page this also flows into the development aspect of it that we're always looking for people to contribute we have a lot of community contributed drivers for storage as storage pieces already trusted interoperability is another thing that we have a very mature CI CD pipeline of what we built into this so being able to test everything for the entire volume lifecycle so if you are bringing your storage or your storage underlying platforms to this that we can actually do all the automated testing for you as well I we also have a package inside of here called Lib storage black people have heard of Rex ray they might not know that lib storage is actually the guts of Rex ray that holds all the pieces for the storage orchestration and the storage lifecycle Rex ray is actually the the northbound interface into say like doing the docker vol driver registration we have multiple platform support which I'll talk about on the next slide this is also it's completely storage agnostic in the fact that we support everything from block file and object so any application that you have that you want to throw at it we can support some sort of storage platform underneath of it it's effortless in the way that we can actually get this up and running we have a very very simplified architecture so we have things such as a curl bash install as well as a docker container install to actually get this up and running and I'll show that here in a minute as well we have a standalone and decentralized architecture so you have architectural choices with this as well so if we think about this and if you go and you ask somebody else and you say I've got we'll say I've got a thousand nodes running inside of Amazon and I need is I need to utilize EBS volumes all right well how are you how are you providing your access keys or you're providing IIM profiles well some people just they'll throw their access keys in a configuration file or an environment variable and now you've got that thrown across a thousand hosts in your environment it's a huge security hole right now of course we can do that as well if you want to go that but we also have lip storage inside of it which comes with a agent control or a client-server architecture so we can do is we can specify R x-ray as a lightweight can client and those can go on a thousand nodes and then we have a centralized Rex rate controller that actually holds all of your credentials access keys whatever it is that I need to talk to my back-end infrastructure so it removes that huge security risk by having all that on all your hosts that are running so here's all the storage platforms that we actually support today right you know we're from Dell EMC you would think like oh yeah well we support stuff on Prem oddly enough when Rexy was first developed all we did is we wrote the ADA you AWS driver so it was since then we've progressed a long long way so we support things from EBS EFS and s3 Dell EMC on Prem so everything from cloud to on Prem to local development so cloud you know digitalocean which was actually contributed by the engineers at digitalocean they wrote it into lib storage we've got other community drivers which is fitted cloud which is an optimized EBS instance we also have as I said for local development if you want to utilize VirtualBox I'm here doesn't use VirtualBox that's a better question right so everybody here pretty much uses VirtualBox well how can you utilize this well what you do is you spin up the soap API on a separate terminal window and I can utilize local volumes on your own host right so you don't need to sit there and spin up a blanket storage or storage back in to be able to mess with local persistence right you can actually just utilize volumes with with VirtualBox so it's really cool and being able to do that we support as I said everything from the development of production lifecycle so if you want to do these Rex rain development a VirtualBox utilize testing resources QA resources in any of these clouds and then something like on Prem with Dell EMC or CF we support all that sort of stuff from from development to production we have cloud native interoperability with all the big players that are out there today so if you're looking you know you're competing and try to take your beating but a lot of us are in the exploratory phase trying to figure out well which makes sense to us as a docker swarm as it mazes as a kubernetes we support all these we have integrations into all of them as well so but today we're going to focus on docker as well as our main piece so we can integrate with docker in a few different ways we can run it as a staple service on every single host we can actually iris try is this a stateless service and every single host to enable stateful applications just utilizing the docker engine we can also integrate with docker swarmed and make it into a robust service in R X ray if you're not familiar with some history we actually started this at when it was the experimental version it was introduced in 1.7 and that branch is actually when we started development of Rex rain so it's been two two plus years now that we've been developing rex ray to the point that it is now I I know you can't see this because it's covered but deployment as I said is super simple there's a simple curl bash command that you can install on every single host and you can throw environment variables or a configuration file at it you say Rex ray start and then it goes ahead creates a Rex ray service on the hosts registers itself as a docker volume driver to the host and then you're off and running second you can actually utilize as we've been it talked about the keynote and this morning the docker managed plug-in ecosystem so with the dr. manage plug-in ecosystem and actually if you go to the start of by docker plugins today you'll see that there's 11 docker plugins for storage in the market today and there's five wrecks raised there today so we almost had 50% of all the certified plugins that are in the Dakin store so that's an easy way to be able to do it and I think I'm gonna go ahead and jump over to the demo now and hopefully this doesn't break because it the bed when I was trying earlier say what yeah right so what I have right here I'm utilizing a vagrant environment that I built and all this is doing is a three note scale i/o environment and it's gone ahead and it's configured a swarm cluster for me so the first thing that I need to do is I need to go ahead and install Rex right so everybody's try to stay off of the Wi-Fi for like a minute right so the only way to be able to do this is I got it I got to pull it from docker hub it's gonna it's gonna install it on this first toast I'm gonna let that go and since this is gonna go pretty quick I'm gonna go for these other hosts as well all right so this is this is this is as easy it is to just get started utilizing directs right here right so it is installing it from the the docker store I'm granting all the permissions for basically all the encapsulating I'm also specifying environment variables that I need actually talk into my back in infrastructure this is utilizing scale i/o in this instance now don't think of scale i/o is like oh I have to use this like Rex rate is a very abstracted tool so I could be utilizing this to talk to installing AWS or EVF with an EFS or EBS or whatever it is right so my well that's not supposed to look like that okay so that one installed will let this one install so as I said I'm doing that I'm setting the file system type to ext4 but only because of this environment we also can utilize XFS I'm also setting the flag for Rex rate preemption actually having high availability set is true all right so now that I've got that I need to go ahead and create some volumes so I'm just gonna for the sake of time I'm gonna copy paste everything over here so I'm gonna go to my my swarm master here and what I'm doing this is I'm utilizing the docker volume CLI or the just the docker CLI to create these volumes for myself so I've gone ahead and created these volumes now these volumes as I said earlier I would show you the the package that we're gonna do but I might give it a way of what I'm getting at here but you might see it in the very bottom I could show you like you know postgrads and like some tables falling over but that doesn't doesn't sound very fun right so I figured let's show like a Minecraft demo so let's go ahead and let's spin up this service real quick and again stay off the Wi-Fi because it actually has to download a jar file so we're gonna say duckers service PS mcc where it's going so it's throwing it over to our tiebreaker machine so it's gonna take about 30 seconds for this to kind of get registered so we can see it's currently in a preparing state i've got about maybe 10 more seconds to go until we get into a starting state alright so we are now running less than a second ago it's gonna take a second to kind of scrape it and we'll find it and once we find it we're gonna start tailing the docker logs to be able to kind of see like where it is in this process now as you may have noticed like with inside this docker service command I know it's a little hacky of looking at everything but we're creating one replica because with inside of that one replica you know we can't break the rules of block storage we only want one container having read/write access to that host or sorry right access to that volume but we're utilizing the mount commands and with this I'm specifying multiple volumes the targets that are actually inside the container and then the source volumes that we had just created and specifying the volume driver itself the cool part about that is that if I had multiple different endpoints to say oh using EBS in DFS inside my own application I can utilize that that tag there at the very end and say the volume driver is Rex ray slash EFS versus the Rex ray slash EBS and so I can have multiple mounts inside of my container to do those pieces alright so docker logs - follow and we'll do a fade here and it the bed so well let's see if we can just do Postgres then not post-grad because it looks like I have like one minute anyway so we'll see see if we can do this so sometimes that happens I don't know why but it looks like it may be I'm pretty sure there's not an s3 outage going on again but sometimes it's just that's just how it works I'm not too sure why let me go ahead and just stop this real quick remove that so I had this just sitting back here just in case and the other cool thing about this is that you know I can create volumes manually beforehand or I can just do it on the fly just give me one second here tap-dancing this is fun right all right so that one's done that one's done all right let's kick this thing off here all right so we'll say docker service P SPG all right so this is going again to the tiebreaker machine and I can say docker PS docker logs - follow 3-0 ok so now we're up and running with our Postgres server I'm gonna say docker let's see what was it okay so docker exec - IT and we'll go into 3-0 Sh and we're just gonna create a database real quick I'm getting the hook I'm getting the hook promise okay so there's our X ray database I'm gonna go ahead and exit out of here and I'm gonna say so when I do this I'm stopping the docker process I'm just simulating a failure here so once that simulate that once that failure goes it should go ahead and just kind of kill it and we will be able to see this get restarted another node so we can see it's already failing over to MDM to come over here docker logs follow be not I almost done a promise so we're up and running I can say so it as you can see the log file was a lot different so it's resuming state you can say docker exec - IT what was this as benign Sh and there's our x-ray database right all right everybody round of applause for me that was awesome right doctor but uh thank you so much about that that ended up working but if you have questions so code le MC booth is right over there it's the container that has spray-paint written all over it come talk to me about Rex Rey and any other kind of persistent data problems you may have thanks y'all

Channel: Docker

Views: 10,111

Rating: 4.5238094 out of 5

Keywords: docker, containers, database, value, store, java, data, swarm, automated, application, Community Theater

Id: U8Dsi5V-XG0

Length: 19min 3sec (1143 seconds)

Published: Mon May 08 2017