Kubernetes in production - blue-green deployment, auto scaling and deployment automation

all right good morning okay let's so let's get started we're going to talk about Cuban Henderson specifically about automating Cuban edits so we're going way beyond the basics although we are going to look a little bit at basics but as soon as we get over that if you start out looking in the AI and now you can actually use that to integrate Cuban edits in your build pipeline and stuff like that my name is Paul I'm from the Netherlands I work for a company called luminous and I wrote one book about osg I'm currently writing a book about Java 9 much so you can kind of see the team about modularity here so you might be wondering why is this guy talking about Cuban ettus which is something entirely different over the past year most of my time has actually been spent working on an infrastructure platform based on Cuba Nettie's and building tools mostly in golang around Cuba netis so well that's right and besides that I'm still writing this Java 9 book so that's how I spent most of my days your writing building software or weightlifting that's my big hobby but let's get started so first of all why should you care about Cuban ettus in the first place um well we've all heard about docker and all of you are using if you're not using docker you're probably in the wrong talk if you use docker you know that docker is mostly about running containers on well your own machine on a single machine there's so networking there and they're getting better at it but it's all fairly limited and there's definitely not something to actually run containers in a clustered they care failover to schedule I think much there is to take care of networking and storage and stuff like that although as I said they are building more and more stuff into docker itself to what to do that kind of things but if you really want to run containers in production you need something like cuban it is to actually make that work just having docker and your rocker command line is just well really not enough very first going to look a little bit at the basics of Cuban edits just to get a concept straight that you have to understand before we can start looking at the API and use this API to really start automating things so if you look at a Cuban ethics cluster it typically at least has the following things it has a master nodes and this master node is kind of in control of the whole cluster the master node decides where containers get scheduled it actually does the scheduling and very importantly it runs the API server and that's what we're going to use later on then we have a whole number of cuban ettus nodes and these are your worker notes so these are the notes where your containers will actually well run and get started the cuban a this doesn't actually just talk about docker containers or containers in the first place but it has a abstraction on top of that which is called pots and a pot it's kind of an abstraction on top of this this container and you will get a little bit deeper into that in in a moment so if our worker notes our worker notes just runs runs a whole bunch of containers and the master is in control of actually making it all work then we also always have a HDD cluster HCD is a key value store and all the well the information about the cluster a store in hcd hcd is fault-tolerant so you Bob you're supposed to have at least three and notes for that and that's where all the information is stored and that means you can actually take down parts of the cluster even the master node without actually losing any state and we're going to make use of @cd ourselves later as well so when we get a cumulus up and running and we have these components that we just saw we can actually start thinking about scheduling from our containers scheduling means actually starting our containers on the cluster and if you start doing that the first thing that you need is a replication controller and this replication controller is an object that runs on the on a server and the replication controller gets configured with the number of replicas that it should start and we will see some configuration in a minute when we configure our replication controller that it should have for example five replicas it will will start five containers from five different of our worker notes of course assuming that we have five worker notes let the replication controller will also do besides starting those containers is actually monitoring those containers if if one of my containers and gets an out of memory exception and just crashes the replication controller will notice that it doesn't have five containers running anymore but they're just four and it will restart my container on one of the other available notes so what gets failover more or less for free by just having this replication controller so this means that when working with cuban edits we are not starting containers on a specific nodes but instead we just configure the state that we want we want to have five replicas of a certain container that's described in a replication controller and a replication controller makes makes sure that you will get to that state and that we keep in that state the replication control is a very important concept then I already just mentioned that cumulus doesn't actually talk about containers directly we're not starting containers directly that we are talking about pots and a pot is just an abstraction on top of a container first of all this is important because cuban ettus is not just about docker containers but it could also support formats like rockets for example which is an alternative to docker this is not actually true yet because currently if you look at cubing it is well the only supported format is docker but in the future there will be different formats supported as well so that's why they have this abstraction another reason for this abstraction is that a bot can actually contain multiple containers so it's not always that a single container is a single pot or the other way around but a pot can actually be multiple containers well this is not to have a let's say different components deploy it together because the life cycle of the containers in a single pot is completely tied together they come together and they go together so if you have let's say a micro service deployments you don't want to put all your stuff in a single pot that's the wrong way to go to it but it is useful if for example we have a simple web server that we want to start we could take a simple nginx container not modify it so we just take an out-of-the-box nginx instance but for an for nginx to be useful we of course have to pass in our our files or HTML files and you can put those files in a separate container just put those containers together in a pot and that very well they can they can work together there's two containers can work together as if it would run on the same physical machine so they will see the files on the same file system they will both see each other on localhost and it kind of works like being on the same physical machine you don't do that that much in many cases there is a one but that's just a single container for a single pot but this is a interesting feature another thing that we get with a pot is networking and of course if you run many containers on a single physical machine where physical machine could just be a instance in the cloud of course we were in many containers on the same machine and all those containers well want to use from ports let's say we have three Java applications in they all want to use port 8080 and we would just run that on the ports on a physical machine well you ran into port conflicts so you would have to do all kind of port mappings that makes things well harder to deploy and harder to understand so wouldn't it be nice if every container that we start gets its own virtual IP address where it gets access to all the ports available without ever thinking about port conflicts exactly what cumulus does and every time we start a pot it gets infertile I P address that IP address is just there for that single pot and it gets access to all the ports so we don't have to even consider port conflicts and not the only thing running on on that IP address is well whatever that container does which makes things very easy but of course then we also get a new problem because if you have virtual IP addresses that will change every time we restart a container because that's that's what happens how do you actually use those containers let's say we have two different pots that have to communicate with each other but if their IP addresses keep changing all the time that's going to be tricky so to fix that problem giving Attis has another concept which are surfaces and a surface is a kind of a proxy on top of your of your pots with a steel virtual IP range but it's fixed so this IP address doesn't change all the time so when you want pots to communicate with each other they don't communicate communicate directly to each other but they communicate using a surface using this fixed IP address and service for just round-robin the traffic to the different replicas that we have so this is very important if you have inter component communication in having this concept of surfaces it also makes it very easy to have well multiple components deployed which all are part of a bigger application maybe you could say Microsoft architecture although I really don't like their micro services each component that you have could be deployed as a pot they each have their own life cycle so you can deploy them independently of each other they have their own namespace because they're all their own IP address and your own port range and using surfaces taken just variously communicate to each other produced as an example because could have a front-end pop that's our our user interface it uses a number of back-end services for replicas of our back-end surface you see there's multiple parts there so we have multiple replicas but we just use the surface to about proxy to those to those instances and we don't really notice it when there are failures because cumulus will take take care of restarting our pots when there's a failure and we don't really notice it from the front-end perspective enemy these back-end surface use a bunch of other surfaces and the ole ball use the surface concept to communicate with each other this works extremely well so if you have an architecture like this Cuba now this makes it very easy to actually deploy that you don't have to have an architecture like this but you can and then the last concept that you have to be familiar with our namespaces in name space is kind of an isolated space in Cuba Nettie's where you have your BOTS and replication controllers and surfaces with Isolators within that namespace and that means that you can have for example a test environment in a separate namespace in your production environment in a different name space you can have a surface with the same name for example without getting conflicts between those namespaces so it is very easy to have different environments on the same physical accumulators cluster right then let me get into a deployments and and this is just the first step because it is obviously not going to be good enough if you actually want to run in production because if you download cuban edits and you look at the tutorials you see that when you want to create a deployment well you have to do that using the cube CTL tool command line tool very pass in configuration files to actually well deploy something and although these tools getting better and better from a user perspective well I don't really want to type anything when upgrading a production application because I'm very bad at typing and I mess stuff up so this is not that we're going to look at dressed talk but I want to show this about the show that's actually really not enough but I do want a show is this configuration file that we have to pass in because that's our first step towards the API so we use cube CTL create and pass in a mol file and the GMO file could look like this is actually something that would work so any GMO follow the we define a replication controller and there's a few important parts here the first important setting is the number of replicas that you want to have so in this case I configure to have three replicas which means that we will start our pot on three different machines in our cluster and more importantly to replication controllable moni's or those spots MN or so failures we'll make sure that we restart and well always get at three replicas then in the bottom there's the actual put specification which describes what container I want to start it describes the actual docker image that we're talking about we can configure about 41 Expos we can configure storage and a whole bunch of things that we can enable for our our pot here so in this case you just want to use the nginx docker image everyone to open up open up port 80 and then finally we have a whole bunch of metadata like labels and those labels are very important when you start looking at the API because labels are used in in communities to loosely couple the different concepts like replication controllers and pots together so a but doesn't know what replication controller created that that pot and a replication controller doesn't have a list of pots that is created but instead both the pot and replication controllers have labels and those labels will match and it can just do queries which pots well kind of belong to the replication controller and this is very loose coupling but it gets a lot of flexibility when actually working with the API and automating things so that doesn't have to be totally clear right now that labels you will see all over the place whenever you do anything with the accumulators configuration or the API so a very quick demo how this actually would look like if you would use cube CTL and after that we start digging into a load balancing and using the API so we have cube CTL get pots and it shouldn't give me anything because it didn't run in anything yet and I have some confirmations file concretion files here so I've got a demo file here exactly the same thing as we just saw on slides I'm not going over it again but we can use this configuration file to say cube CTL create based on this file nginx controller and if you now go back to the get pots we see that there's one pot running all right right that's that's good um but it's only one pot so let's see if you can actually scale this replication controller we have a bunch bunch more replicas so I say cube CTL scale RC nginx that's the name of my replication controller replicas is 5 and we see we are now in a container creating state and a few seconds later most of them should be running that is how we can scale the number of replicas the number of pots that we start on our cluster we can also look at for example the IP address that was created and right here at the top we see that we have this IP address in this IP address is our fertile I P address so there's not an IP address that you would typically use unless you just want to connect to this specific and container but usually would use the surface that we could also create on top of this so another interesting thing that you can play or play around here if you look at this IDs I'm just going to kill one of them and just see what happens we've specified that we want to have five replicas that's not going to be changed so I'm just going to create kill this one cube CTL delete but this ID and if you now look at it get pots again we see that there are still five powers running or actually actually say again there are five pots running because Cuba does notice the failure well I just deleted it so it's not really a failure but I just destroyed it and because my replication controller is still configure to have five replicas the replication controller notices that well there are no five because anymore so it will start a new container so there's a new ID as you can see it's a different ID then in the previous list and that way we got failover more or less for free well of course it was a bit stupid if I just deleted myself and wait for it to restart but the same thing happens when a container crashes for whatever reason if I may if my java process goes out of memory I will get a restarted pot this works so well that we have a few we've had a few production scenarios where our java application was misconfigured it has a didn't have to write memory settings it would go out of memory every few hours and we didn't notice it until weeks later that's not good but it kind of shows how well this whole mechanism works because give me this kind of well takes care of it so this is really nice but as I said this is by far not good enough to go to production if I go to production I don't actually want to use something like cubes ETL and well have the chance of actually making five typing stuff but before we start automating this stuff we have a bigger problem to solve because I've got my nginx container running just now but it can't actually access it access it from the outside world of course we have a five web application that's only useful if I can actually access it from the Internet's that's kind of the point so how do we get access to our nginx containers from the outside world well for that we have to do some load balancing or we have to put a load balancer in front of our cluster so what the want in the end is we want to expose our accumulated surfaces to the outside world and this is not something that automatically works this is not out of the box just support it and a few other things that we might want is things like SSL offloading and setting up redirects and stuff like that things that you would typically use a load balancer for well there is a new feature in QB notice it's in the latest version of queue netis 102 that's only a few weeks old and that feature is called ingress and ingress is about configuring load balancers so this is a new concept in the API where you can write a plug-in to configure well whatever external load balancer that you're using unfortunately it's not completely done yet it's currently just an alpha version of this whole concept which means the API isn't finished and practically speaking it's just well it doesn't really do much yet there are some some demos how it could be working in the future but it's not working today unless you're working on Google Cloud engine because they have support for their load balancers that works great but if you're not on a Google cloud engine like myself then well you have to build something yourself for for an hour at least so in the future this will probably get easier we will get ingress report for multiple load balancers people are working on that but that's probably at least months months away for being available so how do we solve this today well what we do is we take our H a proxy load balancer and just put it in front of our of our cubanelles cluster so here's a proxy doesn't actually run inside communities which we put it in front of it on a different machine or set of machines then of course we want to configure H a proxy dynamically and automatically because we don't want to well start changing the configuration file manually every time we deploy a new pot that kind of defeats the whole purpose so um we have to set up some tooling to actually make this happen automatically and just to be clear we use a proxy but you could do the same thing with with nginx or Apache or whatever load balancer you have because well even with the tools today I will show that could easily be integrated to discount a picture we want to work towards - we get our HTTP traffic inhale a proxy is just running on a separate machine or cluster of machines we are not in communities yet and this H a proxy just looks at the at the address of the request it will forward the traffic to our cube Aneta surface thus Cuba native services of course just our proxy to our actual pots so that we don't have to take care and don't have to track our fertile I P addresses of the replicas that we have they will just forward the traffic to our accumulated surface and accumulate the service for forwarded traffic to the dynamic IP addresses of our pots this is very fun to end up in case you're using something like Amazon like I'm using you know that Amazon also has the concept of load balancer the elastic load balancers and the elastic load balancer shop Amazon are very good at doing the SSL offloading and do it in a very scalable way so we want to use that so in our case you actually have two levels of load balancing we first get all the traffic into the Amazon load balancers to the SSL uploading their forward traffic to our H a proxy which doesn't really do that much B sighs just routing to the correct humidity surface and the Cuban Air Service will take care of actually getting getting to our to our containers another thing that you can do if you are in something like Amazon is actually run the whole cluster inside a VPN that makes the whole security story a lot easier you don't have to do SSL between all the different components inside the cluster that's not that necessary that at that point but you can run everything inside the VPN and the only thing exposed to the outside world is our hedge a proxy and that just makes it a lot easier to confirm all right so conceptually this is pretty easy but how do we actually get it to work because we still will have to configure our hedge a proxy to know about our accumulated services and if you look at hedge a proxy and the same is true for something like nginx they still use static configuration files so if you want to add a new back-end to hedge a proxy and a back-end in this case means our queue Benetti surface which knows about our pots we would have to change this configuration file reload H a proxy and well only after that it will work and of course we don't want to do it manually so we you say little open-source tool called conf D and confuse us a tool that watches hcd hcd again it's our key value store that is also used by criminals internally but you will just write our load balancer configuration in sed use conferences in LCD and every time there is a change we will use a configuration template to regenerate the configuration file vanish a proxy trigger I reload and everything is automatically working again that's map may sound sound like quite a bit of work but actually company is an extremely simple tool the only thing you have to go give it is a is a template and this is pretty trivial to to set up all right so we kind of solved the first part you can now actually access our applications from the outside world and now we can start using this whole setup to actually automate things and we probably want to automate to the points where we can do a git commit and make sure that that gate commits actually ends up in production but of course within those steps within those deployments we don't want to have any downtime so that's the important step you want to work towards now when doing any kind of automation cuban alias you want to start looking in the api and actually if you start working with cuba netis i think the interesting thing about cuban ettus is actually the api out of the box i would not say that cuba netis is a well tool that you would use out of the box of course you can and of course most tutorials are are showing that that you're using C cube CTL for example but I think the much more interesting part of communities is actually the API so that you can actually make it part of your bigger picture make it part of your old tool of your own toolset and that works very well using this API using the API you can literally access anything in cube netis cube CTL itself is also just using the REST API so you can get access to well everything that you would want to do in Incubus it just rests so you can use any kind of language in any kind of tools to to work with this API but to make life a little bit easier there are a few client libraries two of those libraries my team wrote one for Jaffa and OSGi and one for go the reason for that is that when we got started with that there were actually no good client API so there was one for go but at directing off the internet on dependency so that was not very useful so we wrote some client libraries ourselves the java van is completely based on Rx and there go version is just well go as you would expect from go they're both open source the party licensed and I'm Dow to umbrella a project there's client libraries for different languages as well I don't know the quality of those but there's a whole bunch of them okay so animation we start automating production deployment specifically we have to be very careful that we don't have any downtime during deployments looking at at our own teams we do production deployments in the middle of the day when there's well people actually using the applications so it's pretty important to actually have zero downtime and if you want to do zero downtime deployments you kind of have two options you can either do a Bluegreen deployment or you can do a rolling update it's a rolling update you say that I five verticals running and I'm going to upgrade the version of whatever is running one by one so I'm going to upgrade one of my replicas when a pond is finished I will grade the second one etc that works pretty well but the downside of it is that people have two versions of our application running at the same time during a deployment and that makes it a little bit more difficult to actually deal with well that that time window which are two versions running because if you upgraded your user interface to use a newly introduced API on your back end and the back end is only there on a few of your servers start fulfill so it makes it more difficult to deal with the stuff from a coding perspective that's one of the reasons we like Bluegreen deployments ourselves and that's what we will use for this automation so in a Bluegreen deployment we will make sure there's always only one version of our application actively serving loads at a given time and the whole mechanism looks as follows you get traffic to our application so heishi proxy will receive some requests and if it is forward forward them to the replicas that we have running and we currently only have a single version running of the application so that all makes sense then we start a new deployment and I just a new component the deployer and we will look into that in a moment and this deployer deploys a new version and deploying a new version means that it spins up a new replica set so we'll get a number of new replicas running our new version of the application but as you can see H a proxy doesn't even know about this new version it doesn't put any loads to the to this new version yet because the first thing that we want to do is you want to start health checking this new set of replicas that we have you want to start health checking if the new version is actually well ready to serve load so all those components running here should support some form of health check and this can be a very simple health check like just doing an HTTP call and see if you get a 200 status code this could also be much more deep health checks where you actually check your database connections and all kind of connections to other services and if you have a dynamic environment check if everything is started correctly and configured correctly so you can make this quite quite complicated but only after all the replicas that we have report to be healthy only then be a right configuration to convey reconfigure head shape proxy and from that moment on we will serve traffic on our new version and the old version can just disappear we can just delete that one this whole mechanism is not queuing at a specific the concept of blue-green deployments have been around for a very long time and this is always a trick using a load balancer you just instruct your load balancer to first of all use the first version and at one point it does just decide all my traffic is now going to my new version there's not a new concept but it works very well with with Cuban Eddy's so as I said I just introduced a new component the deployer because we had this component that did some health checking and took care of reconfiguring H a proxy at at the right time so what is this whole thing that we call the deployer but this would just be a concept that we talked this is not something that is reported inside communities it is just from tooling we write around Cuba Nettie's I will just show you the concept how you could actually build that yourself and I'll also show you an open-source tool that kind of does it for you so let's deploy our that we have here this deployer the first thing that it does is well it creates a replication controller using Cuban Etta's API and the cumulus api on the stern will start my pots and it will create a surface for forest deployment and that means that but we just spin up a you could say a completely new cluster for our application then the deployer will notice that those replicas are started and we'll start doing health checks so that could just be an HP call on slash health so that's something that we configure in a deployer we can configure a a pot and a port for example for doing a health checks and if as long as the components actually support those health checks that will all work out when all the pots report to be healthy we will write some new configuration at CD as a set s CD is the key fail you stored store use back units internally but we will also use it ourselves to store this load balancer configuration because it's already there anyway so why not use it so we will write some data to Ed CD and this a little tool comedy it's just watching at CD for new values and whenever it finds there is a new value in s CD it will create a new configuration file for head shape proxy and it will give this new configuration configuration file to add a proxy restart head shape proxy in a way that it doesn't drop any connections you can configure that in head shape proxy and most other load balancers and from that moment on H a proxy will serve all its loads to our new store it represents our new application version as you can see this is this is not very complicated there's a bunch of stuff going on but it's not very complicated there's only a few things you have to do in both the API and the communication with HCD but it's not definitely not rocket science you can actually write as yourself quite easily not sure why this is in there again but as I said there's also an open-source tool that does exactly this the steps I just described there is the deployer that we use ourselves we figured well why not open source it because there's probably useful for a lot of other people so this just takes you to truly a step to do this a Bluegreen deployment it takes care of this load balancer configuration by just writing to 2x CD so if you don't want to use H a proxy which we support out of the box you could easily plug in a configuration file for for for another load balancer this whole tool is written in go Y go well cubes itself is written in go show it seemed to make sense and it works quite well besides that it was just a lot of fun to actually do something useful and go and it's a partial licensed so if we start using this version of the deployer we get kind of a deployment descriptor that we can work with and this is just well a few few of the settings that we can use there one of the things that we can enable here is a use health check flag so that way we can say we actually do want to do those health checks for this component during deployments well why would you want to disable that let's say you have a container that you do want to make part of your deployment but it it's it's a third-party component and it doesn't do any health checks well then this is your way out if you configure the number of replicas we configure a front-end URL that's the URL that will be configured to inhale a proxy and at the bottom we have two BOTS back and that's just a bot spec that we get from cuba netis so this goes back to the cumulus api that we configure our ports and our images and stuff like that so it's quite simple and we just pass this in as Jason then there's another tool though kind of part of this this open-source project is deployments ETL and deployment CTL is a user interface to configure your deployments you definitely don't have to use this if you use the deployer that we just talked about because in the end is just a REST API but it just makes it a little bit easier to configure your deployments and redeploy stuff without actually touching any kind of API so that's quite nice as well and that's written in a combination of products and angular 2 I've got a bit of a demo for this and because there's some darker pushes going on etc I actually recorded that so we don't have to wait for that the applications we are going to deploy is a simple go application and don't worry if you never seen go it's extremely simple because well this is the this is a whole thing what we do here we create the HTTP server that only responds to slash health so this application kind of only does a health check nothing else and if we invoke this slash health and point the thing that we get back is a struct just a JSON structure containing a version of the application which will be set by the deployer a message which is configured at a string at the top and a boolean if the component is healthy or not and I'm kind of cheating a little bit here because the boolean is just set as a fixed boolean value right there in the top but that will give me the possibility to actually do a unhealthy deployment later on so that's that's why we start playing with that all right so let's start looking at this we will first check if there's any deployments currently so we'll see cube CTL again to actually see what's happening under the hood there's no pots running and the front-end URL that we will be looking at doesn't respond to the moment which is good because it means there's nothing deployed yet this is a deployment CTL the user interface that I just saw about because configure the front-end URL the number of replicas and but of course docker container that you're going to use you get completion on a number of things there we can set all kind of resource limits which are also important in QP Netta so as you can well run as much as possible on the same physical hardware you can configure ports so we have our health check port that we will look at em let me just click deploy you see the locks and in this case it's configured in a way that it will actually sleep for twenty seconds before actually reloading the configuration in H a proxy you know now if our pot running so we see that we have a replica and now we can just wait for those 20 seconds or actually well the waiting time is cut off the cut out of the movie and whenever it's completed you can look at the URL and we see that we have the first version now deployed just using this user interface ok that was good but let's try a second version so we change the message of of the application people do a bolt of this image and actually push it to the docker registry and it will simply click on deploy again we see that the version here at the top is automatically generated so that's why we just had version 1 and we will now see version 2 it's spinning up our new version but again so what you see now is that there's two versions running next to each other but our front-end still gives us version 1 why the load balancer is still waiting for the health checks etc to happen and it didn't switch versions yet after 20 to 20 seconds because it's configured like that it does and if you go to the curl again we see that it now gives version 2 and the old version of the of the pot will be cleaned up by now it is pass another healthy deployments now let's try what happens if you do an unhealthy deployments I messed up medic coding mistake or configuration steak and a push something to production which doesn't actually work but luckily we have our health checks because what's happening now is that it will go through the same steps it will spin up our pots if you go to cube CTL marks you see both versions running sort of pots for both of them but in contrast to the previous deployment it will time out if you see well this health check is never going to get healthy so deployment filled and it actually roll back this deployment and we will just stay at the old version the new version is to just delete it the nice thing about this whole mechanism is that even if you have massive amounts of concurrent requests coming into your application they will not actually notice any downtime during those deployments we are actually testing this whole mechanism using Gatling that's really just firing as many requests as possible to a version and we will see no drops requests so that means that our users don't actually know or care if you do a production life update if it fails they will just stay at the old version no harm done if it succeeds we will go to the new version then there's of course one thing that gang can go wrong if our health checks actually succeed because we have bad health checks but the application crashes as soon as we put actual load on it well then we did write some really crappy health checks to begin with but that would be a problem because then well everything will just collapse after the deployment is finished and then you will just have to roll back manually well whenever you finish but when you find that problem but it doesn't typically happen a lot all right so that's what we can get with the accumulators API and using if you want to from some open source tools around that and this also makes is very easy to actually completely integrate this in your build pipeline just now I've been using the user interface to actually start a deployment but you could also completely automate that and just look at a welcome it to master for example our build server will build a new docker image they'll push it to docker hub or whatever other dr. registry that you're using docker hub supports web hooks so we can just invoke a web hook on the deployer the deployer knows which deployment that belongs to it will starts the starting new deployment as we've just seen and well we have a completely automated deployment to potentially production and that that seems to work quite well all right it's a completely different topic scaling but you have already seen in Cuba this just now when I was starting to play with cube CTLs that keeps ETL as the scale command and with this scale command we can very easily scale the number of replicas within the cluster that we have well that's very great but now in practice we don't just need to scale the number of pots would be probably also want to scale the number of nodes and in some case you can scale up but maybe at night time we actually want to save costs to to Amazon and scale down the application as well and that that doesn't just mean that we can scale the number of pots because even if well one of our nodes not doing any anything we would still pay to Amazon so we actually want to scale the number of machines that we're using as well and that makes it makes it a little bit more complicated because let's say that we are actually running an Amazon or some on a cloud provider yeah cube netis doesn't really know about that so it doesn't have the constructs within Cuba Nettie's to take care of this machine level scaling it just takes care of the scaling within the cluster and we have to take care of both again using the API we can quite easily do that to scale up its Rea easy we use the Amazon API and let's assume we're using Amazon there's API for the other cloud providers as well of course and for Amazon we can use a scaling group to configure the number of cubanelles worker nodes that should be active and we just use clout in it or other kind of strips that whenever a new machine is started in the scaling group it initialized itself as a cube another's nodes will register itself to the master and it's good to go wellsure flowed from that point on you can just scale up the number of replicas that we have and they will probably start on this new newly started machine so that's easy scaling down just a little bit more work because then we have to make sure that we don't actually destroy a running application by not turning off its its machine so the first thing if you do there is actually set a note to an scheduled ball that means in cubanelles terms that it will not actively remove anything that runs on this server from that server but it will not start any new containers on this note the next step is to drain the notes and that means that we really relocate the running pots to different notes assuming that we have enough capacity and only when this note is not running any of our containers anymore we actually remove the note from Cuban Eddy's so for Cuban ettus it doesn't exist anymore and we use the Amazon API to actually well shut down the machine and stop paying for it again using those api's you can easily do that but we have a longer open source tool for this and that gives you a CLI to actually go through the truth our steps and probably the most interesting part of this is that there is a way to configure schedules so just using this JSON format you can configure that at in this case 9:00 in the evening you'll switch to half of our capacity and we will both configure the number of replicas for applications that we have but also the desired capacity of our Amazon cluster using this schedule it will automatically scale down an evening and probably you'll have another schedule that actually scales up again in the in the morning and that way you can well have a scalable cubanelles cluster and pay as little as money as possible to do something like Amazon this also works works quite well although this whole concept is not supporting Cuba netizen the scatter to point using this API you can kind of make Cuban letters do whatever you want so I'm actually going to skip this one more point to make this is a question that many people ask and they will ask well there's all great but what about my database because my database obviously has a state and how do you run something like that in a dynamic environment like Cuba Nettie's because well Cuba netis as you've seen so far is kind of stateless it just restarts containers periphery at once and it doesn't get quite work well um - OH - that is currently you don't don't run data stores in Cuba Nettie's technically you can and you will definitely fight find tutorials on the internet to do so for pretty much every data store out there but you are making your own life very difficult cuban ettus does have the concept of persistent storage that all works but you actually configure that correctly and tune it correctly to run a well highly available data store like or my sequel or whatever that's actually not very easy and that's mostly because data stores don't really support this whole concept of the offer of a dynamic environment like this they don't like auto scaling and things like that so just make your own life easy don't run your data stores in Cuba netis just run them as you previously did on whatever server that you want and let your Cuban ettus services use those data stores alright um ever set my team is working on a cumulus based platform with monitoring and logging and all that kind of good stuff stuff that you would definitely need there's a lot of good tools for that we just use all kind of open source tools to do things like a movie during escrow fauna for example to definitely look outside just to scope Cuban Edison you're looking at this kind of stuff and well that's that's what we do a lot of time thank you for your time if there are any questions you have one minute left and after that just find me in the hallways thank you any question so the question is why not use a community service instead of my head a proxy load balancer that's a good question because we kind of have two levels of load balancing now a community service our first is still on a virtual IP address you can expose it to an actual port on a host machine with as well well kind of make the failover story a little bit more brittle because but if you restart the machine on Amazon and it gets a new IP address for example because that's dynamic as well that's why we put as a proxy in front of it so we get a more reliable maybe not the right term but a more static configuration that's forward stuff to to communities that's that's the reason for that any other questions yeah or very good question again and this always comes up how do you take care of scheme chances for your database when you do a Bluegreen deployment as I said that's a very good question and that that's that's another problem that's easy to solve if you have especially a relational database and you have this kind of blue-green deployments no matter if you're using Cuba nettles or not scheme changes are going to be difficult so you either have to be very sure that the scheme changes are always backwards compatible so that your new scheme also works for your old scheme then you can first do the database upgrade and then do the application upgrade that's a possibility you can do hot swapping of databases I know some people who actually boot up a completely new database cluster with a new scheme and just roll over that that whole database instance that's an option if you have a scheme most database like you kind of move it to your code level because of course there's a scheme somewhere in your system but that makes it a little bit easier to just take care of that from your code but it's it's always a difficult question it's an unsolved problem I think I'm now really out of time so if there's any more questions just come forward and ask me thank you very much

Channel: Devoxx UK

Views: 93,898

Keywords: DevoxxUK2016

Id: -Ci4vd4rh4M

Length: 53min 10sec (3190 seconds)

Published: Thu Jun 16 2016