To Crd, or Not to Crd, That is the Question - Ed King & Sam Gunaratne, Pivotal

all right everyone let's get started so thank you so much for attending our talk and on custom resource definitions also known as CI DS it's great to be here in Shanghai let's start off with some introductions so my name is Sam gonna Radner I'm a software engineer working for pivotal and from London my name is Ed King I work alongside Sam also from pivotal and also from London great so there are a lot of talks about C IDs at this conference it's sort of a hot topic in the community at the moment so let's just set the scene about what we'll be talking about in this talk specifically so ed will start us off just talking about our motivations for giving this talk then since this is a beginners talk we'll just cover some basics about the kate's api customer resource definitions and operators then we'll move on to talking about what it means to build on top of the kubernetes api but things that are not workloads based on kubernetes and we'll talk about the pros cons and some of the considerations for using coop the kubernetes api but some things that are not so obvious all right so I'd like to just take a few moments just to run through some of the motivations that we had for flying all the way out here to China to come and talk to you today and so I hope that this will help you to understand exactly where we're coming from and also to sort of better frame the rest of this talk and so this all started about six months ago when Sam and I were assigned to a new team and we were tasked with building a brand new system component that would eventually integrate into part of a bigger product that we were going to be shipping and so we went off we did some research we spoke to some users and we eventually came up with an idea for what we might like to build for our alpha and the question then became okay so what technology should we use to build this alpha now traditionally we probably would have just reached out for a classic web server probably written in golang with a backing relational database and that would have been great however at the time we were very aware a lot of noise coming out of the kubernetes the kubernetes community around the kubernetes api and specifically with regards to extending the api by a use of these things called c RDS and so are a little ears pricked up and we thought okay what might it look like for us to build this system component on top of the kubernetes api as to as opposed to a more like traditional web server with a backing relational database and so we went off again we did a lot of research we spoke to some of our colleagues we took a look at what was going on in the community and we learnt quite a lot of stuff and so really the idea for this talk is we just would like to go through some of our learnings and perhaps some of the pros cons and considerations for what it might look like to build something on top of the kubernetes api so that if you are yourselves asking the same question like should i be doing this hopefully by the end of this talk you'll have a better idea as to whether or not that's something that you might like to look up further ok so let's take a quick step back and talk about the kubernetes api from a high level so the kubernetes api is an api centric so what do I really mean by this so when I first came to communities and I came from thinking about wanting to deploy workloads won't use communities as an Orchestrator and a scheduler for my containers but the more I came to understand about kubernetes the more I realized that one of the reasons that it is such a good scheduler is because it has this API with a set of rich features that allows for the kubernetes ecosystem to evolve and four features and api s-- to be added an extreme rate from companies all across the world and its really the kubernetes api that facilitates this so let's have a little look at some of these high-level properties now the first is of the communities API is declarative so this is the Watts versus how approach so in an imperative procedure way you might say I would like to go to node a and start one pod and then I'd like to go to node B start another pod you're telling the system how to achieve something with a declarative system you say what you want to happen so you say I won two pods and by the way don't have those pods running run on the same note and there's been an evolution over time of how you treat infrastructure moving from procedural to declarative so you might write have written bash scripts ten years ago to manage your infrastructure and then you might have chosen something like chef or ansible to manage your infrastructure and then things like terraform came along which is a purely declarative way of defining infrastructure and cuba dennis takes that to the next level being declarative but also as you'll see constantly reconciling that state of the world though next is the the cuban a's api has this clear separation of states so there are two pieces of states one is the desired state that is specified by a user or another system then the other is the observed state of the system so you have this clear separation there's uniform across the cuban api and that allows for ease of reasoning about the system next up is that the cuban a's api is level based rather than edged based so what do i mean by that this is terminology that sort of comes from system programming and electrical engineering and it's about how you measure the changes on a wire over time so in a H based system you notice the individual changes and you keep a track record of those changes level based however you just periodically poll and check the value and so level based systems are more forgiving for distributed systems for example if your component crashes in an edge based system when it restores it has to understand the history of all the changes that have occurred over time whereas a level based system just reads the current value and makes it so so really it allows for less error-prone components to be written and as I said it's more forgiving so next up is that the CUNY's api is transparent by default so writing resources to the API you have a single control plane for everything and permission allowing all of your components can access all the other resources from other components so this allows for composability between teams controllers can watch other teams resources and then extend the functionality all right so those are some of the high-level features that make the kubernetes api really really interesting what I'd like to do now is just dive down a little deeper and walk through a typical example in order to take a look at some of these core components we're going to need to know about in order to understand the rest of the talk and we're going to go with with the classic example of like creating a pot and I'm sure that some of you probably very familiar with this so I try to keep it keep keep it quick I just want to make sure that we're all on the same page so that we have the terminology understand the rest so at its cool the Kuban Eddie's API is a restful api backed by an HDD datastore and there are a bunch of resource objects that are available on that API so these are things like your pod node deployment and these are the objects that users of the API are able to interact with and to create update and delete and as a user of the API it's up to me to declare my desired state of the world as Sam said the kubernetes api is entirely declarative and so for this example i might say something like I want one pod to be running and the way that you do this is by posting gamal into the API and typically this is done with a cube CTL apply - eff take a quick look at that Yama we tried really hard not to get any Amal into this presentation but that is a really hard thing to do like when you're talking about the kubernetes api so we're try to keep it brief the important points to note here are the kind in bold so here we're specifying the kind of resource object that we're interested in and then also the spec where we are declaring our desired configuration for that object and so we take this yeah Mille and we post it into the API and it goes and gets stored in the sed data store and so at this point I need to introduce the next really important component in the kubernetes system and that is the controllers so the kubernetes api is comprised of a number of controllers and each controller is watching for any changes to certain objects of a certain type so for example this controller in the top left may be looking for any changes to any object of type pod and when it notices a change in this case that a new pod has been created it gets notified about it and it then goes through a process that's known as reconciliation so essentially what happens here is the controller will go and look and grab the desired state of the world so in this case the new desired state is that we want one pot to be running and it will then go and look out to the current state of the world in this case that's 0 quads are running and it will then take whatever actions are necessary in order to combine those two worlds together such that the desired state is equal to the current observed state ie that one pot is running and then once this has been done it will talk typically talk back to the kubernetes api to update that original object to update its status so it might say something like status state running true or whatever it might be and this is really important because this is how other components that are interested in this object are able to find out its current state so we've got our resources we've got our control is how the custom resource definitions fit into all of this well essentially custom resource definitions or see IDs for short allow users of the API to define their own custom resource types so perhaps I don't want to create a pod or a deployment perhaps I want to create something more exciting let's take an example of a lovely piece of cake everybody loves cake what if I wanted to create update delete cakes on my kubernetes api this is what's the ideas allow us to do and so in this example we might say ok actually I want a kind of cake right and in the specification that I'm saying that I want a Victoria sponge type of cake and the nice thing here is that I can take this this declaration this yeah Mille and post it into the kubernetes api in exactly the same way and using the same tooling as for any other resource that would exist on that api and so i could post that off it gets stored in the HDD database but there's now a problem because by default none of the default kubernetes controllers know anything about my cake types right and so at this point nothing's going to happen and what this means is that as me as the author of the cid i also need to bring along my own custom controller and plug this all in and this new custom controller will be watching for changes to type cake i will know how to reconcile the world upon any changes to those objects and this is really at the core of the extensibility of the kubernetes api using crts and your custom controllers and so you could imagine that you could keep going building up your own custom types custom control is such that it looks a little something like this ok so let's have a brief look at the timeline of extensibility in the communities API so around I think communities 1.2 third party resources are introduced and this is the first time that you had a way to add new things to the kubernetes api so shortly after core OS introduced this thing called the operator framework and so really what they realized is the Kuban is is great for running workloads especially like you know stateless workloads but when things have state and a stateful you need to concern yourself with the day to operations like taking a backup keeping the thing running making sure things have quorum that sort of stuff and so korres developed this framework that allows you to extend the kubernetes api to run stateful workloads and a great example is there will be an operator for my sequel and so that operator is job is to replace a human operator that might treat it have traditionally looked after that my sequel cluster this operator will ensure the cluster is alive and healthy and take backups and that sort of stuff and this has proved an immensely popular pattern in the committee's community so the operator framework is one framework of actually doing this but now there are more such as coop builder so in 2017 CRTs were introduced as a replacement for third-party resources to have even closer alignment to the kubernetes native resources and to be more powerful and then after that we started to see bigger systems being built on top of see IDs it's a good example of this is Kay native which is an entire new platform and set of abstractions on top of kubernetes another great example is sto which allows you to get an entire service mesh on top of kubernetes and I think sto has something around 23 or so custom resource definitions that it's made up of so next this is something that was very interesting it was the AWS released an operator and this was special because it didn't act on resources within your cluster instead it acted on resources in AWS is public cloud so as a developer I created a custom resource asking for infrastructure and the operator would go to Amazon's public cloud to provision that and this is an interesting example of what the operator pattern being used to not manage workloads in your cluster but managing something completely different and so this really got us thinking like what is the next step of this transition Percy IDs so just in the near term series are hoping to go GA in the next couple of releases there's this movement towards and desire to want to use see IDs to replace some of the native resources in Corki binaries so yeah we wanted to spend the rest of this talk thinking about where this might lead cool so let me paint you a picture so for the next part of this talk we're going to be running through an example service and take a look at what it might look like to build something on top of the cube kubernetes api and the example that we're going to go with is that of a home automation service so we're going to imagine that Sam and I have a hugely successful home automation company and we've got a really great home automation service that's used in loved by thousands of people all around the world and before taking a look at what it might look like to build that on top of kubernetes let's just imagine what it may look like today right and perhaps it looks something like this perhaps we've gone with a classic microservices architecture whereby some user of the system posts some JSON into some front end some gateway and the micro services go off and do whatever they need to do so let's take an example here or perhaps this is what it might look like to switch on some lights in our kitchen we take this Jason we post it in the micro services go off and do that thing and because we're a hugely successful company we have hundreds and hundreds of developers co-located all across the globe right and so it's quite likely that each service is going to have a corresponding team who are going to be responsible for developing that service and really each team their main focus is that they want to be delivering the absolute highest quality best lighting service that they possibly can right and so let's let's draw some team boundaries around each of these so that it looks a little bit like this and then we can also imagine that each service is going to need some data and so each service is probably going to need a backing database right and so it's going to look something like this and then we're going to imagine that we really care about a high availability and the reliability of our service is really and really important if this goes down users are going to be locked out of their home and it's going to be a disaster so let's add in a whole bunch of high availability to each of these services but we know that nothing is ever perfect and things go wrong and so when things do go wrong we're going to need to make sure that each of the services has a whole bunch of like retry logic baked in and then finally we're probably also going to need some sort of like service discovery mechanism so that each of these services can find each other in order to do the job and so actually like I look at that and I think that there there's a lot going on there right and importantly there's a lot of stuff going on that actually is distracting each of the teams away from delivering the features on each of the services and that's really the thing that they want to be doing right so there are a lot of distractions each team has to think about their storage the backing databases and all of the operational overheads that come with that the high availability the reliability the API contracts between all of the services all of this is just taking time and effort away from the main focus of each team which it's developing the best service that they possibly can and so maybe maybe there's a better way okay so we very well could have written this service with these micro services and then kubernetes would have been a great tool to deploy these services into we could have created deployments and pods and run this service in kubernetes however given that we now know that you know the community API has all these great features could we actually build our service on top of the kubernetes api so not using kubernetes as a container scheduler but just using its API to achieve this sort of business problem that we have here so let's go through that example now the first of all each of the team wouldn't be responsible for creating a micro service like a restful micro service there be responsible for creating a controller instead and we wouldn't have a gateway instead we'd have our controllers watching a single kubernetes api and so before our teams had to manage their in database now they can rely on the kubernetes api to store their data via C IDs so in this world the user might express intent to the system declaratively saying I want the kitchen lights to be on and so they could post this into the kubernetes api and they might have a resource that represents a room for example and I have a specification for what I want the desired state of the room to be in terms of the lights here so you can see I have I want two lights to be on with different brightnesses so that could be posted in and we would have a controller the team that's responsible for that watching the API it would be notified it would start to reconcile that request it would look at the system and say hey there's no lights on so what would it be it would create some light resources and those light resources would represent the individual actions for each light bulb in our room wherever it is so the light resource just has a single speck requesting the brightness and then this would be picked up by our lights controller and this is the boundary between our two teams in this example the room team just creates this light resource event it's another team's responsibility through its controller to satisfy that request so the light controller then would go and look at the actual state of the world go and talk to the LED bulbs wherever they are notice that they are not on turn them on and then update the status in the kubernetes api you can see here at the bottom it's set the status to reflect the spec because it is successfully reconciled the desired state and now be observed state matches but then our team's controller would be notified of this change because it's a parent of these light resources it created them it has an interest of what's going on with them and then it would update the room resource with the status of the lights and this is important because the user or whatever system component created the room maybe doesn't care so much about the internal resources used to accomplish the request it cares about this abstraction and so it can just look at the room resource to see the status of the request and obviously in this world if one of our controllers goes down for example then it's decoupled from the API you can still post room resources post light resources and then when the controller comes back up again it can just reconcile and because of some of the aspects of the API being level based that will just work all right so let's perhaps perhaps fairly I could like contrived example I'm not suggesting that you should immediately go out and rewrite a home automation service on top of the kubernetes api well maybe I am let's see if you are thinking about doing this sort of thing there are going to be some pros cons and considerations right and so for the rest of this talk we're just going to be walking through some of the main things that popped up during our research for this so I'm going to talk through a few of the more purely sort of technical considerations and then Sam's going to talk through some of the wider considerations around this as well I'm gonna kick things off by talking about storage and really here I'm referring to this idea that if you are developing on top of the kubernetes api you essentially get the backing at CD data store like for free right it's just there it's part of that API control plane and it means that it's probably owned and operated by whoever is responsible for keeping the convene Eddy's API up and running and so as a developer who's developing on top of that API this is hugely enticing to me because it means that I no longer need to worry about any of the operational overheads that are associated with managing your own database but of course there's like there's no such thing as a free lunch and there are some other considerations around here it's not quite that simple one of the one of the main things that popped up while we were sort of looking at this is that HCD is not a relational database which means that some of the relational features we've come to know and love unavailable and the example that we tend to give is what if you needed to do a big join of your data right it's just not really possible and so I guess the key the key thing to note here is like just make sure that your data model is suitable for running in this new model recently readily related to the storage is this idea of high availability so when deployed in a multi master configuration the capes API is highly available and again that's sort of highly available just for free right again really really enticing to me because it means I don't have to worry about the availability of my API it's just highly available for free and I then have more time to focus on writing my services and shipping the features that I care about there is another aspect to this this idea of high high availability I think Sam touched on it earlier when we were talking about high availability of the controllers themselves and so there's maybe a question of like hey should I be running my controller is highly available and I guess I guess the thing to note there again is that in this like level based system perhaps it's actually okay if your controller goes down for some period of time because all that's going to happen when it comes back online is it's going to read the desired state of the world trigger it's reconcile function and away you go and the last thing I want to talk about is performance so again the performance of your API is going to be largely dependent on the underlying EDD datastore and there have been some concerns around the performance of EDD in the past I guess the key thing here is or maybe the key worry is that if you've got a particularly noisy service that then could potentially have a very negative impact on the performance of not only your system but also like any other system that is also running on that kubernetes api and so we've got to be a little bit careful I guess the general advice is if you if you're developing something that's going to be slamming the database quite a lot like maybe like a metrics thing or if you're doing something like dumping your logs in the database probably not a good idea but we do also have this this idea that there's a there's a big conversation going on in a community at the moment as to whether or not you should just be running one huge kubernetes cluster or if perhaps it's better to run lots of little ones and if you're off the mind that lots of little ones is the way to go then perhaps these sorts of considerations are actually not so bad it's even possible that you may just have a separate dedicated cluster purely for these like extension you may not even enable the scheduler one man for example so there's lots of things to think about the final thing I want to say about performance is the API machinery folks are thinking about scaling targets for CID as themselves and there's a survey there if you're interested please go put that out so if we want to take advantage of all of the benefits of kubernetes being declarative eventually consistent for our own needs of our own service then we need to put in some work to actually achieve that one of those things that we need to do one of those things we need to do is we can't just take our micro service and turn it into a controller very easily it's a completely different paradigm of writing software your service it's probably like a restful service and it's request and response talking to a database may be talking out to some other services through RPC or more rest calls however writing a controller is running a reconcile function that is watching an API and converging state it's a very different way of thinking and writing software and as I just said saying not everything fits like the cumulates API is opinionated and for good reasons but if what you're trying to model doesn't fit so you do need that request response synchronized synchronicity you can't map your domain as resources or you have a very complex domain model then it's going to be an uphill battle and you're going to be fighting the API to model it in kubernetes next up and this is particularly interesting for us that what we really want to do is collaborate the way the kate's community collaborates we want CRTs to be the standard interface between our teams in the micro services example we should saw earlier maybe each team had to define their own API and maybe you have company standards for that API we also had to have a service discovery system so that we could discover all the other API is available so in kubernetes the CID is the interface between our team and what's the ideas are available in the cluster is the discovery mechanism and so this interest EDA leads to interesting collaboration patterns where we can have different teams responsible for different see IDs and maybe independently another team can monitor the status of one team and extend the functionality through this mechanism so it allows for real loose coupling of interactions between teams and finally we don't really have time to go through too many of these but there are a whole bunch of other API features that you probably want to consider and research if you are thinking about doing something like this I just took through a couple so binary data for example you should not be using sed to store to store binary data if you do need this this might not be the right approach for you alternatively you could upload your data to something like s3 and then just refer to it in your specification in your custom resource next if command any supports defining relationships between resources and there's sort of two main mechanisms for doing this the first is a owner reference which allows you to set up child parent relationships between your resources and this gives you properties such as if you delete the parent the child resources will be cleaned up the other common one is adding labels to your resources however it should be noted that managing the integrity of those labels is up to you as the controller writer so you have to ensure that those labels make sense in a changing system next versioning is interesting so custom resources allow you to have multiple versions installed at the same time so you can imagine that teams are writing new versions of their custom resource but other teams are relying on older versions so there's some backwards compatibility features there and now in communities 1.5 there's a feature that allows you to migrate from one custom resource version to another custom resource version and finally since we have this one control plane the kubernetes api and everything is represented as c ids we can take advantage of any tooling that exists for this control plane so couple will work any you eyes that are developed that will work the way you list everything in your system will be ubiquitous and so all tooling will be the same so I guess the question is sort of remains is it a good idea to use C IDs outside of the scheduler so to see I do not to see ID so if you are wanting to run complex workloads that maybe are stateful on kubernetes the operator pattern seems to be the emerging technology and way to do this and as C IDs get more features they're going to become even more applicable to other problems such as the home automation system we showed here and I think we'll see maybe an explosion of experiments in this way in the next sort of 12 months to sort of see what things can we build with the kubernetes api itself but just be careful of jumping on the bandwagon when any new technology comes out there's a whole exploration of what is possible so I hope this talk has sort of helped with some of these considerations and things you should be thinking about yeah and so I think Mike one of the main things that we learn in in looking at all of this over the past six months or so is that people have very strong opinions about this right some people think this is a great idea and you should go all in and it's gonna be fantastic others are more concerned think that just stay away and so whenever in doubt you know we'd like to look to Twitter to see what people are thinking there was actually like there was a good tweet the other day there kind of some some of this up which was this one here and I guess the takeaway is like yeah this is a new technology I think people are getting excited about it and I think quite rightly so I think there is a lot of potential in this new way of working and I'm really really excited to see where it's going to go but at the end of the day we need to remember like not everything has to be a CRV and with that I think that that's all we've got time for so thank you very much I don't know if there's time for questions perhaps if anyone has any or perhaps just come speak to us afterwards oh yes [Music] so it was a very interesting talk thank you so I have a question so if I'm making a cluster wide ceoddi and if I want to how only one instance of that CRD on my cluster is that a pattern to do that it's like a singleton CRD I guess the honest answer is I'm not sure so there's quotas which have been introduced in a new version of cabinets but I'm not sure that cost of white coats is I don't know if anybody knows I'm so your shakes have head so I don't think so think so you can set quotas within a namespace but not a cluster wide but not today you know there are questions is those CIP controller only works deploy on master node or my question is how those controller scale yeah yeah so actually like your controllers don't even need to be running in kubernetes so for example like when we're developing this control is one pattern we see is that you run your controller on your local developer machine and you just have to point it and make sure that you have access to the master API so you can actually run these controllers from anywhere I think the typical pattern is that you would typically run them inside the cluster but there's there's nothing that says that you have to do that yes so I have a kinda change about the mountain version and we all see IDs I'm not sure in your case how are you supporting the mountain version for USAID for example for your case for Team a he wanted the lightning service version but example version one but they added every team C wants this version to how managing the relationship between the mountain versions and the other one and how you validate your say at least gamer how you make sure all the same object it's correct answering your second question first and so you now have structural steamers for your customer resources so you can ensure that the customer resource has the correct structure but then I guess it's up to your controller to do validation and you can do this file like an admission controller so that you can reject requests before storage if it doesn't breaks any of your like business sort of logic that sort of stuff question again okay so I guess you're always gonna have to have communication between teams so the question was and how do you manage having multiple customer resource definition versions and multiple teams that rely on different versions so I think like you series allow you to have multiple versions install but you're still going to have the problem of needing to know to some degree which teams are consuming that version because you don't want to keep your old CI DS around forever as a team you want to know when you can deprecated them and I don't think there's any sort of technology solution for that it's sort of a team collaboration sort of problem so so you mean for example I have tuition our live service so if I have no way to force that he may only use the version one but only by the collaboration between team it's the case using the newer versions in communities and you can change you can migrate from one version to the other but the other team has to be aware of that so that you don't break them or you have additive C IDs so that they can still rely on the older versions so I want to know if there any way to fast an example the team a only rely on question one but Tim being reliant version 2 is there any way to do two for this no not I know of in terms of forcing different teams to use different CID versions and I don't not that I know okay thank you all right well thank you very much everybody

Channel: CNCF [Cloud Native Computing Foundation]

Views: 6,657

Rating: 4.9148936 out of 5

Keywords:

Id: xGafiZEX0YA

Length: 38min 5sec (2285 seconds)

Published: Fri Jul 05 2019