Kubernetes Operators: Managing Complex Software with Software [I] - Josh Wood & Jesus Carrillo

all right good afternoon everybody I'm Josh wood from core OS I'm here with ACS curio seniors officer senior systems engineer at Ticketmaster we're going to talk a little bit this afternoon about a concept we've developed at core OS or a pattern you might call it for management of sophisticated stateful complex applications on kubernetes and in terms of the kubernetes api we call that concept operators if we do a good job this afternoon I will help to introduce some of the general principles and ideas behind that give a couple examples of those that we've developed ourselves or that we see in the field and then I think the really fun part of this will be when hey sis tells us about their use of a particular operator at Ticketmaster to run a particular piece of mission critical software in their overall services stack so welcome cloud native con coupe con 17 and thanks for coming on how many folks here verdict over Nettie's I know you know the thing we all love the most about kubernetes are certainly that drew my interest to it and I think is drawn corporate interest open source developer interests and certainly industrial interest in the platform over the last year a year and a half since open sourcing is how easy it makes it to scale stateless applications if we're talking about something like a web front-end or any simple app that doesn't have connections to persistent storage that doesn't have its own Nessun notions of clustering or internode communication scaling applications on kubernetes is as simple as using the tube puddle scale command to scale a running application say from one to three how does this work well we refer to the parts of kubernetes that provide command and control lifecycle management as the control plane and that name comes from the primary piece of software that runs in that control plane which we call a controller the basic stock kubernetes controller has a simple job it looks at the desired state of an application or deployment described by Emmel manifest file or wise coordinated by an administrator compares it to the actual state of that deployment or application running a cluster and takes actions or steps to reconcile those two states this in essence is the controller concept you could call it an Oda loop you can observe act and make decisions to correct situations so let's say I have two pods running for my simple web server application I just issued a coup puddle scale for three replicas essentially the controller loop looks at the difference between my desired state in the actual condition in the universe and adds the pod so that now I have three we've even made this simple enough that you can do it with a graphical application in our tectonic console and take a look at the number of pods running ask for one more and you can watch it come up running live here in this sort of animated slide so taking a look at the cube scheduler just ask for two two are running very simple stateless applications but what about apps that you know store data I some of my applications tend to read and write files or report things in a database or have other notions of connection that expand beyond what these kinds of stateless apps expect in the world and are less easy to migrate between cluster node members in the event of application failure or scaling requests like the one we just looked at we'd really like it if managing complex apps like databases and file systems were as easy on kubernetes as creating them is and is easy to manage and scale those apps as the simple stateless app that we took a short example look at it's not too hard to say who cut will run some database you got an image and you bring it up and it's running however what if that database has its own notions of clustering in terms of masters and slaves or leaders and followers where are the files that record the persistent data for that database located in terms of + or resources how do the pods that implement the database application find connect to main authentication or otherwise access those staple resources when these items exist in stateful applications when there's a notion of clustering within that application like in database or distributed file system apps how do we resize or upgrade the members of that cluster without disturbing their own clustering connections to one another while still doing it in terms of the kubernetes api coop cuttle and the other tools we use to manage kubernetes secondly how do we reconfigure these kinds of applications when it's not enough to tell them make one more copy of this how do we issue commands for locating persistent data stores persistent special network connections perhaps even access to particular computer resources last but not least any of these complex apps with complex data that persists somewhere in the world either an object storage disk some kind of a volume claim in the end that data needs to be backed up and managed with its own notion of what a life cycle of data preservation looks like all of these things are beyond the knowledge of the basic stock kubernetes controller in the control plane and are beyond the features presented by the stock default kubernetes api if only kubernetes knew about complex applications so the answer to make that happen is to take advantage of the extensibility designed into the API and the kubernetes architecture to extend kubernetes to work with particular applications the goal is to come up with something simple that extends the kubernetes api to let us talk about things we can't currently describe in API terms today what's the size of a cluster of databases or file servers that I want to have running on my kubernetes cluster how many of those members are read replicas followers or slaves in their own notion a clustering system what version of the software of the data store format is my distributed database or distributed files or using while running on my kubernetes cluster operators are a way to encode this specific application knowledge from a human administrator or operations team into software and pair it with extensions to the kubernetes api in the form of third-party resources to allow kubernetes to manage these complex stateful applications in terms of the kubernetes api while at the same time doing this in an extensible modular way that doesn't disturb or interfere with or add complexity to the basic primitives that make up the kubernetes architecture I think the easiest way possibly to bring this into some kind of real picture that you can reason about is to take a look at one of the original operators we initiated at core or less as we begin to define and explore this operators pattern how many of you have heard of SED so really like it so hard conference ask questions to write like like how many of you are familiar with the diet of worms in 1331 and you know like you have to change subject matter or everybody raises their hands sed is a very popular distributed key-value store it solves a hard problem in distributed computing and does it in a simple lightweight way that human beings are at least somewhat able to reason about by implementing the wrapped protocol rather than talk size for consensus in distributed databases in short what sed does is store little bitty pieces of information that need to be coordinated and kept consistent among multiple members storing that data around a cluster you find that CD at the heart of kubernetes storing cluster application deployment services metadata for the stock kubernetes control plane STD is also an important member of many other popular distributed applications from our friends in the open source and industrial communities that's why this will sort of all on that slide that's why ICD is one of the very first things for which we try to build an operator so to review our basic operator ideas operators build on key Nettie's concepts like the controller the control reconciliation loop and the API itself which it extends through third-party resources and TP RS extend that API with new API objects and you can think of these kind of like a database table schema like this is the data model for this particular special application that we need to manage with an operator extensions added to the API through third-party resources includes some of those samples that we were looking at in our idealized vml description of an SED cluster manifest cluster size cluster version 4 sed now of course the TPR mechanism was designed to build exactly these kinds of extensions onto the kubernetes api so with the SED operator we built an operator that knows how to manage SED clusters and keep them running reliably over their lifecycle at scale deploying this operator looks really really simple on the surface you pointed at a yamo file and at the end of the talk I'll give you the URLs where you need to find this all of the work I'm speaking about today is open source so you can try this out on any kubernetes cluster pretty much immediately this actually deploys the operator itself rather than an sed cluster what happens after this step is we have now added the third-party resources and the custom controller binary in the cluster that are then able to deploy create and manage sed clusters for us again looking at sort of a idealized deployment for a net CD cluster these extensions may not be numerous or even in a great amount of depth they just provide us the keys we need to make API manipulations of an application around which the API was not originally designed so we're looking at how big is our cluster what version of s CD should be running in that cluster pairing those extensions to third-party resources with our custom controller we have a replacement or augmentation of a human s CD operator that knows how to deploy STD clusters by deploying a master writing the initial data store adding new member nodes and connecting them maintaining the sed client endpoint aware applications access sed services how to backup sed in a data lifecycle management sense and also how to perform spot backups before automatically undertaking operations like restorations of failed nodes RIA ditions of new nodes to the existing sed cluster and maybe most interestingly upgrades in versions both of the sed binaries and the sed data store format between different versions of sed so to take a look on the screen here if you look at sort of Group one we have we have what the controller maybe would consider the picture of reality the thing we are going to compare to our desired state well if you've looked back at the Yambol we were showing our desired state is considerably different from the state represented in block a here we want three pods we only have two running we want them all to be running SED version 3.1 point zero but we actually can see that of the two that are running only one of them's running the version we requested so just as na standards kübra a standard kubernetes controller but with additional specific application knowledge the sed custom controller that comprises with the third-party resources the sed operator can examine the difference between reality desired state and knows how to actually take steps in sed specific terms to bring the condition of the sed cluster to that described in the desired state that we showed in the manifest file what's that really going to mean in this particular highly simplified idealized illustration it's going to mean that we're going to first of all recover at least one member because we're short of one member of the three we've actually requested that run in our cluster we are going then to actually back up the data store shared by that entire sed cluster because the next and perhaps most important step we're going to take in our reconciliation is upgrading that recalcitrant 3.09 member to 3.10 in short that's what the SED operator does those are the basic kubernetes primitives it extends to accomplish that work and it is open source code which you can examine use to run your own sed clusters at scale on kubernetes manage them in terms of coop peddle commands in the kubernetes api and additionally the sed operator is currently one of the best and cleanest based examples of where you might start writing your own operators in addition to SED like we've seen several operators arise both inside pour OS and in the field first of all I'll talk a little bit about things we either are working on right now or have just delivered in the sed operator and what it means to to some of our product and project offerings one of the things we talked about a lot of core OS is this idea of self driving self hosted automatically updating infrastructure core OS container Linux rather famously automatically updates with a sort of a/b rollback partition scheme and a really similar way to your Chromebook or your mobile phone we're actually able to deliver Linux operating system upgrades rather than being bound by dependencies we can deliver these upgrades because the container gives us isolation between OS and application layer so the same I to the same end we are trying to drive this idea of self updating auto updating self driving infrastructure up through the stack from the operating system to things like at CD and to kubernetes itself the operator work is a key underpinning of how we're delivering self driving infrastructure and delivering kubernetes clusters that can upgrade and manage their own life cycles over time rather than requiring manual or administrator a activity at grade points the sed operator lies very near the heart of how that work is being delivered if you look at the latest release last week of tectonic 150 5.2 shortly before the kubernetes one six release and in our blog post making the release announcement for this we talk in some detail about the more high availability at CD version that was delivered in this last release a lot of that is built around the the concepts of operators and how they manage and automate the management of SED um and now I am going to talk about one of the other operators that we built in the team and another open source operator for managing the Prometheus monitoring and alerting system how many folks here used Prometheus to monitor their systems supposed to raise your hand hisses I so Prometheus is a really neat monitoring system do a lot of charts and graphs we can take many many detailed measurements and much of the kubernetes api and the sort of pod interface and applications that you run on kubernetes are instrumented in ways that Prometheus can gain insight to and report statistics and measurements from out of those statistics and measurement you can construct reliable meaningful alerting systems for when things do go wrong and people have to take action or perhaps alerts the fire operator actions in an idealized automated world before I take too much time I am going to point out that our Prometheus operator is like the SCD operator open source work that anyone can use and roll out one of the folks that has taken a lot of initiative in rolling this operator out learning it in deep detail and building production systems with it at scale at his place of employment I happen to have standing on stage with me today so I would actually like to turn this over to KC's carrillo from Ticketmaster who's going to take the general outline of what operators might be what they can accomplish and drive them down to the practical level of what Ticketmaster and hey sis's teams are doing with the Prometheus operator at Tech for today it is okay good evening everybody um emphasis Korea I work at the kid master and today I'm going to talk about our journey on deploying and using Prometheus ticket matter so first of all I want to set the context here and I want to tell you how to get master is going to use or is using Prometheus so as we transition to a DevOps model we are getting rid of on open th TV because it's a gigantic burden in our operations teams to keep huge open th EB cluster alive we are replacing legacy on alerting systems and now each team is going to own its own monitoring and alerting which is better that way so we we started evaluating our primitives and we did up remove concept it's super basic and it was closed two teams that arm they had already instrumental applications and making them our Prometheus compatible was uneasy step and teams that were in the process of migrating their applications to AWS and I wanted to take the extra step to make it on prometheus enabled the architecture of this pfg was super simple it was Prometheus and Allard manager running on very few instances and it was sharing which it was shared between teams so the PLC architecture looks something like that we had a federation and the federation is just another set of promises on scraping the arm lower layer promises and aggregating all the metrics into a single point so we can hook up Ravana and do pretty drafts at a single point what what we what problems wink and the lessons that we learn with this POC we were cheating a lot of Federation script amounts because the data set that the arm lower-level our premier cases were getting in was huge so we can get a like a one-minute resolution because the data set was too too big and we were when we didn't have time to get all the matrix by configurations can disrupt the service arm throughout that merge request then had a typo and the new alert rules weren't evaluated for like a day then at the Federation side ticking the storage parameters takes time and it's like trying our error if I increase it the instant size and more memory to the instance I had to go back and tweak those parameters and also on the teams were facing a lot of network ATL's problems or network is highly restricted so we had to find a way to make this um the user experience better what we learn is that each team shall have its own from each attack and we need to divide to conquer so then we say ok um let's roll out Prometheus as a service so the teams can just quickly deploy their primitive stack and get a lot of things out of the box with it so the Prometheus as a service must allow the teams to quickly deploy ax are dedicated stack and I have the goal of making it 2 minutes per meter stack it should not represent any additional burden to the team so they just permit it'll just work for them and we always up to date and also provide pre-configured et2 and keyboard native service discovery based for ec2 based on tags and Kira Nerys service discovery based on annotations and also it should be based on helm because we could roll out our newer versions of the service discovery templates um pretty quickly and easy and then we came up with a exporter a ticket master exported database so to solve the network shield problems we provided a well known port of of a well known port range for the exporters we open the akio and also all this great jobs and easy to service discovery it's based on this database exporter database then up so how we can achieve this well with the Prometheus operator it knows it allows us to model our complex primitives stack in a few lines of code of configuration also for the Federation the storage is outta tune so if I increase the memory limit for the pond that is running our Federation workload the primitives operator does the math and configures it and an important thing is that our manager is highly available by default with the primitives operator I just need to say give me three replicas of our manager and it will configure the clustering for it so what we're looking forward with on this primitive operator arm we we are really interested in the Federation EPR because right now it's manually and also Federation charting because and this is really important for us in a non sale I don't want to be 2 a.m. in the morning preparing for an on sale and have to scale everything and charge it manually right and also graph on integration so if new primitives stack is deployed to the kinetics cluster I can have the arm data source added for it and one interesting thing is the company adoption rate everyone loves it we have two weeks of production already clusters and the adoption rate is huge right now we have like four teams that are running their production grade monitoring and alerting on chromakey our primitives and the parameters operator and we have like 40 teams that are evaluating it in our development clusters and this is a quick chart of of hell or architecture looks we are using tectonic clusters and we have deployed a set of easy to instances that are running there are then has a specific Negra cells to be able to reach the ec2 instances on the port range that we carved for this and then um we are using node selectors to schedule the parameters workload to go to specific nodes and we also have calcium channel or network policy and then the federation stores the data into any beers volume EBS is eating the EBS are provisioning is also managed by the operator so we don't need to do anything there does any one has questions about this a question about the meat department or about how operators relate chart and you're like yeah thanks so the question since I don't know if we have the question mic room in the audience that seems to be sitting up here on the podium but the question was not specific to the Prometheus operator but in general how do operators as a concept or a pattern compared to things like Helms and the other deployment distribution techniques we've seen a rise around the kubernetes ecosystem and secondly like you made a suggestion that operators maybe seem to you more like a concept or a way of doing things so I'll start at the end I think you're right in that sense um the way what I think I like to think of operators is the capital o as a pattern or a concept the way that complex applications can be managed on kubernetes we have not boiled operators down to an API that you can use to easily construct your own I would imagine the future of that and the work that I know about I'm trying to generalize that stuff is - it is probably going to look more like a like an SDK like sort of maybe skeletal sample code things that you begin to build on top of a few functions that are the main abstracts that we've repeated in our development of several different operators for different apps but one of the key things about operators and why we didn't necessarily go directly to the kubernetes api and instead extended it through the TPR mechanism and these custom controllers their very nature and purpose is the long term management of the lifecycle of specific applications they do encode a lot of application specific knowledge that we don't necessarily think belongs architectural e in the main kubernetes api which is why we chose to go sort of this route of extending the api so that the long term how do I build my own operator question that this sort of train of thought leads us to is today unfortunately or fortunately early days all this is pretty new stuff you can take a look at the existing actual operator code at least in the HCDA operator there has been some work to abstract the general operator functions somewhat from the sed functions but they still live in a single codebase right now in the future I would expect there to be something like an SDK a basic software development kit that is like the bottom of an operator the place where you might start writing your own how do operators compare to helm I am here's a I'm going to answer this somewhat glibly and not even make any attempt to hide my ignorance of one side of the question I know a little bit about helm I certainly like all the folks at Deus and I think I've even been out there to Boulder to speak and so like I don't I just don't happen to know a lot about home and use it on a day in day out from my knowledge which is limited the Asus might have a comment after I wrap up here helm is very very deploy and replicate or in it it feels more like configuration management if you map it to an older kind of concept or something we maybe all have shared knowledge about an idea of what we think like config management is operators are a long-running application that stays with these special applications as they run on kubernetes clusters and not only deploy them in easy single command-line ways but manage their overall life cycle can actually upgrade them when an upgrade is available not when you run a set at home commands to deploy a new version in your chart and I think that's pretty much how I see the difference between between like then but like what I know that folks do with helm versus what I do know about operators and what I do myself with operators I could even and here will be where I go too far and really reveal my ignorance I could even imagine a helmet that deployed some kind of custom operator it might handle the deployment and replication of that and over and over again to then sped up the operator which would then handle the actual lifecycle management of the app over time after that deployment yep so for example we're using home to tilt the operator that we need our Prometheus instance right so we we use it to generate like the in the examples that George did on the the Gamo file that gets pushed to the API and and we would help with hem we construct the the GPR configurations and then say to given anything deploy this and then the operator takes it and it knows how to for example create another manager and create a cluster of other managers that that's the difference right not yet because it has a lot of Ticketmaster specific stuff because for example we do or et to service discovery it's based on our own ec2 tags so make maybe we can do something later [Music] which would be which is more something I should and then something happens that there commences you know I appreciate personal that it has an astronomic you a thank you okay so the roadmap means actually did not know about all such an extra that their question on the Q&A they had but I waited to inform my answer because I was trying to do a little bit Lauren conveyed with us refusing I mean I item I don't personally have it must be so knowledge about that plan to bring home into the problem very interesting idea clearly Khloe has you talk more about operators because this questions come up in some degree before home and some mystery that people mine I'm going to simplify the problem and be able to understand tactical ways yes difference and how am i entertaining about [Music] so the question is are there plans for operators for complex database applications like my sequel or Postgres sequel we have done experimental work on them definitely for Postgres sequel that is a piece of our overall infrastructure out at some edges and would be important to us and they are also those are exactly the kind of apps that we would expect to see community development on I know about this much about early stages Postgres sequel operator that did if I since I don't have my slides I'll try to look it up after this talk maybe hands you a URL will shortly afterwards that's the only like here's excellent work and code that I can send you to go look at but certainly those two applications along with applications like set distributed file system storage are certainly what we had in mind as we began developing the operator concept and so to go just a little bit away from your question and talk about that idea of file storage there is a there's a company in a product called Brooke and they are kind of a Enterprise packaging of the Ceph storage paired with an operator based on the operators pattern that can manage set file system clusters in a much more complicated but still basically the same as what I was showing with what the SVD cluster it knows about SEFs notions of clustering so it can maintain those notions of clustering of who's a node who's a leader where's the persistent storage live in terms of pure Nettie's and on top of kubernetes so there is a work operator and it is a usable in live running operator for running largest urban file systems a lot of problems are very similar in solving need the large database systems issue so yes is a very long yes - one of the things we love to manage with operators of our databases databases are tough to managing distributed clusters right now we don't have good ways to manage a Postgres sequel database in kubernetes terms if we basically build a Postgres cluster up on the side and then someone is responsible for maintaining that cluster keeping it alive and so is exactly the kind of problem we'll assault with operators so the question here is the previous talk talked about this idea of an open services API which is sort of how to manage service endpoints is what that sounds like to me so set of commands added to the default kubernetes api or set of end points in that api that let you describe management of those service endpoints my suspicion is that one the idea of the open services api is a general pattern for all applications running on kubernetes whereas while an operator might do very similar work or even itself use such an api to implement its work it will continue to be more application specific for an SED for a my sequel for a Postgres sequel and the thing that distinguishes operators in it I think you know actually in software engineering you try very hard to generalize everything you can and reserve the specifics to the places where they must live part of the idea of operators is they are that place where since there have to be some specifics this is where they should live these custom controllers these third-party resources my suspicion is that the API from the previous talk is much more general and would be a general kubernetes feature whereas you would build an operator for your application that you run your work and that I may not know anything about I think that's how they would differ again I know this much about an idea for for a services API and an endpoint control API so I have an idea what's talking about but I didn't see the previous talk so it a little hard to do a direct comparison certainly what distinguishes operators is their specificity to particular applications that's so what's what's different about them than the work we do every day upstream on stock open source kubernetes is we're trying all the ways to generalize and abstract stuff we're putting in the main code base and reserve the specifics and the intricacies of individual application design - to these operators reasonably good answer I like can you score me lessons this going to add to that Indian forget hey

Channel: CNCF [Cloud Native Computing Foundation]

Views: 10,295

Rating: 4.7391305 out of 5

Keywords: CloudNativeCon + KubeCon Europe, CloudNativeCon + KubeCon, CloudNativeCon + KubeCon 2017, CloudNativeCon Europe, CloudNativeCon + KubeCon Europe 2017, KubeCon Europe 2017, KubeCon 2017, KubeCon Europe, KubeCon, CloudNativeCon, CloudNativeCon 2017, CloudNativeCon Europe 2017

Id: cj5uk1uje_Y

Length: 38min 47sec (2327 seconds)

Published: Mon Apr 10 2017