Kubernetes 1.19: Accentuate the Paw-sitive (RTFM with Rawkode)

[Music] so [Music] hello and welcome hello taylor how are you do you know ryan about yourself david yeah i'm very well thank you it is getting towards the end of my day so it's a a nice way to end the day with a nice look at kubernetes and spend some time with yourself how are you today doing well it's uh yeah just just kicking off my day uh personally here so i promise i know you're at the end of the day but i'm gonna try and bring that caffeine and uh some of that good morning energy uh to the stream today awesome i definitely appreciated that's for sure so you are a developer advocate at hashicorp and very relevant for today you wear the 119 kubernetes release team lead if you want to add infants to that and absolutely so so 119 uh just kind of you know i'll i'll just kind of jump in and out of some topics on that front but uh howdy everyone my name is taylor dillozal i'm a senior developer advocate at hashicorp and um really really excited to talk to you today mostly going to be covering the 119 release and what that experience was like some of the fun new features around that and then just that whole release process honestly david was was was a wild experience uh beforehand i was working at walt disney studios on on the sre team and uh and and you know helping with all the efforts on that front so um the 119 release you know was one i believe the longest release that we've had thus far um between all of the kubernetes releases that we've seen and so kind of navigating jumping between jobs in that all the wild things happening in the world you know it's just been it was a a marathon release is what most of the team had said around that which is uh absolutely true yeah definitely so i i wasn't actually aware that this was the longest release but i guess you know there's been a lot of different factors in 2020 that may have contributed to that i believe you've been under a released team a while now right how many releases absolutely so i started on the 114 was i believe the first release that i was on uh and saw the shadow survey and then jumped in and got started with the communications team so for those of you that aren't aware the release team it is one team but it's broken into multiple different verticals so we have you know one one grouping it takes care of communications another one takes care of enhancements for example another documentation ci signal there are different groups that focus on different things throughout the course of a release and so typically the the release teams in like in the like lower mid 30s i think we had about if i'm counting correctly i think it was like 33 34 people on the 119 release yeah i think people watching this will probably be a little bit surprised that you know with the size of kubernetes as an open source project that they're it requires a lot of people and time and effort to actually get that release out the door it's it's interesting too because you you know the there are different special interest groups within kubernetes but again you know for those of you that aren't aware and they're you know sig re the special interest group just for the release handles that release and then those uh those subgroups but there are other ones like sig api machinery sig docs you know and and even more on that front too so no shortage of people to ping or reach out to on slack if you have a question about a feature that's going to make it or or not and how have you found the release teams now that's if you said 114 was your first this is your fifth release right so you've experienced a lot of the role i guess yeah so i took i did take off the the 117 uh release cycle so that was and kind of got to you know get a good get some uh rest and relaxation uh away from that because it does it does take a lot out of you um but uh really really been fun to see how the group has evolved in the different ways in which we go about communicating with one another and seeing you know all of the documentation and handoff processes get you know more and more mature from release to release um but uh and and honestly it's it's uh in in my perspective my bias it has been a lot easier from release to release and just kind of that you know talking with people and even just you know again it could have just been the stars aligning for me with the 119 team but it just was so easy to work with everyone you know everyone was just so kind everyone was so upbeat and because it's an open source project you know that's you're not there's no financial incentive hence open source uh everybody's passionate about the project and so they're bringing that passion um to work with them every single day that you're working on that release and so that really you know that passion really powered this release that's uh where the accentuate the positive title came from and uh just it was fantastic for people to work with and uh and everyone was just you know even if there was a little slippage here or you know getting some done early or late everybody was just there and it very much had that family feel to it for a lack of a better term that's awesome and i was very curious that they accentuate the positive so i'm glad that we kind of covered that i think every release now that i've at least been visible to or i've been paying attention to it's like the teams are always great and those come from the release team leads right you're the one that accentuated the positive hearing oh absolutely yeah just just uh just trying to do a good thing pay it forward um yeah each each of the releases has a theme that will come with it which is chosen by the release lead um and then typically it's well i i don't think it's ever been disclosed before the release has gone live uh but that's usually kept a secret even you know uh uh not even told to the release shadows as well and so you know sometimes they'll be like hey what do you think you know there's that back and forth kind of workshopping on the idea or you know just kind of double checking to make sure everything uh all the the everything lines up but this one was definitely intended to be a surprise and was a lot of fun uh picking that out um that for for anyone that's seen it it's a uh animal crossing themed uh type logo uh just very much in the spirit of and uh you know that game came out just as the 119 release kicked on and really started up uh and then you know as as we've had to go through covid as we've had to uh you know just injure under all these socioeconomic and other you know natural disasters just all these crazy things that we've had happen to us around the world uh no better time than to focus on one another really just being upbeat uh finding out how you can help one another and just kind of you know just you know joining hands with people and just kind of figuring out how to meet them where they're at and just to be you know good kind uplifting um and i again i feel we really did that in 119 giving people time as you know some weeks were a lot harder than others uh taking a break from meetings um you know definitely around kubecon we took a break there as well but we really focused on the community in this release you know if you over over exert uh the community and the people working on this project you're not going to have a lot of people left to to work on the project if if they're all burnt out right so it was a lot of fun to work with people on that and just really actively listen and see what was going on in the community excellent i just popped up the great kind of artwork there um it's definitely one of my favorite ones i think everybody's been a little bit animal crossing daft this year so i thought it was a very fitting name and i think the sentiment that you just described there as well the release is fantastic as well for great job man now i don't want to get this wrong but i believe that shadow applications are open right now for 120. is that correct that's correct uh i believe that they're closing up soon so if you're looking to uh get your start on kubernetes and interacting with the open source community uh by all means please please please check that out um i'll try and find the link i'll try and reach we retweet that out later try to say that five times fast and uh and share that with y'all but uh i do believe that's closing up so uh and if you have any doubts about getting involved with open source uh if you're on the fence if you're having that that's uh you know if you're if you're undecided on that front uh just do it uh i obviously make sure you have enough time to do so uh but uh there are rules for for everyone and i the biggest the biggest piece of feedback that i've heard there is that a lot of people are concerned about um you know being judged or just they don't really feel comfortable sharing their code or their contributions with the world you know that's that is a lot of scrutiny scrutiny but it's uh you're working with fantastic people and they're not going to you know let anything bad happen uh so long as you work with people you have that community mindset and uh and that's the goal that's the intent that you're walking in with you're you're not going to be disappointed but by all means make sure that you have time to do it um they're they're i think when i selected the communications team and i started on that with the kubernetes release cycles um that that rule fit really well but you know something like ci or ci signal that one is very involved and it requires a lot of communication between team members and so you can kind of pick what works for you which is really nice about the open source project yeah i think just to kind of touch on one thing he said there like you don't even have to write any code so you don't have to be fearful of writing code you can join the release team as a shadow and start to experience some of those different roles and positions and you know never write a single line of code i know i've never written a single line of code with the kubernetes project so and it's still a very positive experience i really enjoyed the couple of the release teams that i had as a part of two and plus you get to meet so many cool people and just all those interactions and and even if you are a little bit shy and you're curious or on the fence you can just go and look in the second release channel you can join the mail unless there are loads and loads of ways where you can get familiar with it first without having to jump straight into the shadow applications but we will sorry no that's it's such a great point david i i think that it's such it's uh most open source projects yeah i i've felt like that where it's just like you kind of jump in you don't really know what to expect kubernetes there's a whole bunch of like role books there's that shadow process so you don't you know it's you're not you don't have to be like uh you know just completely in it and you have to know everything before you do anything that's the whole purpose of that shadow program is so that you can take the time to see how things work and then ramp up until you're at a point where you feel comfortable to take over something in a leading capacity typically we've we've advised that like two cycles of being in a shadow position before getting to that lead position is you know typically how that works out but uh yeah it's it's come come come lurk come ask questions come enjoy the banter and the conversation uh it's not always about tech you know sometimes we will talk about animal crossing we'll pop open a zoom chat on a friday and just kind of talk about life you know like it's uh it's it's it's a family it's community it's it's a lot of fun it doesn't have to be uh there's equal parts work and play uh when it comes to working on kubernetes and that's that makes it a lot of fun yeah definitely okay that's that's great um we will tweet other links i'll make sure they're in the comments section as well in the youtube video the release team is a great experience i hope we've done it just this we've sold it really well there feel free to get involved now we're going to talk about kubernetes 119. it's got it's got some major themes like uh we've got a list here and do you want to read through them do you want to just dive into each one after one what's your preferred format here what would you like to do yeah so so i'd love to kind of uh hit on a lot of major themes for 119 and just kind of you know if they didn't if they weren't on your radar would just kind of love that opportunity to put them on your radar um i i and i'll of course i'll i'll point out which ones are my personal favorites but uh we can do we can just jump through them um so one of the the biggest items that happened in 119 is increasing the kubernetes support window to one year whereas previously it was at nine months uh because every three months typically you have a new kubernetes release and so that allowed for you know just just two back as soon as one was published so now at any given point in time starting with 119 there's going to be four that get supported there um there's a lot of feedback from the community saying that you know it's nine months is it is a sizable amount of time however uh would like a little bit more time so because it is you know it does get a little bit dizzying to have to upgrade your cluster with every single release that comes out and uh and most teams aren't able to do that some can't do that even within the nine-month period time uh so we we hear you we listen to you and uh and we have one year of support so that's that's a lot of fun to see yeah i think that's a change that's going to be welcomed by so many you know kubernetes being such an integral part of people's production infrastructure even updating every you know three months can be very daunting but then getting to the end of a supported release even even scarier i think that extra bit of buffer time will really really help a lot of companies it's it's really going to be fantastic and i do hope that you know my selfish want on on that one with that feature specifically is uh some of the managed kubernetes options that you have you know i think that most of them are are are have 117 uh supported right now but i you know my personal want is to see that a little bit closer to uh what's you know what's gone ga like seeing 118 just a couple months after it comes out would be great to see same with 119 just so that the the community is able to kind of work with those features see see if it's you know good fit for them uh and you know and so on but uh happy to see that you know we have those manager clusters available as well and then for any any of you playing at home uh i'd absolutely recommend checking out the kind project by the fantastic ben the elder um that allows you to spin up kubernetes in docker kind k-i-m-d and uh it really makes it easy for if you want to test something you know work with the control plane or just kind of spin up uh you know like that that's how i i checked and looked at and started working with a lot of the 119 features out of the out of the box out of the container yeah okay kind is definitely the easiest way to start kicking the tires on some of these features especially even mini cube as well has really good support for both of these so you know you can you can have your shot with any and i think one team is available on both pretty much immediately after the release which it just makes it so easy for this oh absolutely agree as far i i don't want to ask you an opinion piece so early on in our call but i'm going to do it anyway and there has been many conversations over the last 12 months about whether kubernetes the release cycle is too fast like and corporations and companies are having trouble to speed up is that something you see changing maybe in the future do you think it will slow down as the project continues to mature or do you think that maturity will continue to encourage new contributors and maybe speed it up that's something i have thought a lot about myself and i think that i personally i do think that that will become the case at some point in time and or you know will maybe will separate out into kind of you know core features and you know looking at how the process is set up right now such that you have you know you introduce a feature with alpha you get feedback from the community and you get you know that conversation started then it moves into beta after you have you know a good backing there then finally ga on that front um so i like that process but there have been a lot of conversations that i've been in both inside of the community and you know and in the end user community and there is you know it's the stability is constantly becoming you know a cornerstone to every release that happens for kubernetes um and being able to you know there's talks about like can we release bug fixes in a different you know just to release that asynchronously along with this core functionality you know will things get broken out into you know different types of providers and you like having core functionality and then other different you know veins to kind of feed that into the project i'm curious to see if that happens and then there was one talk that was given at uh kubecon san diego by tim pepper and steven augustus and they talked a lot about you know we don't really have a distribution of kubernetes out there right now we ship the product but we don't have this distribution with an opinionated like uh almost like an os right we don't have anything like that so i think as time goes on we'll start to see more uh both distributions or just methods of delivery of you know what channels are you following for these features and kind of keeping track of that i'm really really knowing the kubernetes community i know they're going to they're going to do it the right way they're going to talk with the community before anything happens but i definitely want them tracking that it's going gonna be an interesting one for sure all right keep an eye on that one definitely then so we've got 12 months everyone should be happy they've got the time hopefully they need to stay on a supported version and i think with you know i think a lot i've had here's another i don't want to say way far too much again but your previous role to hashicorp you said was it disney right if i remember correctly you weren't sre correct yeah ah right did you run your own kubernetes clusters so mostly we focused on just uh you know disney's a big place so so lots of technology is used but um really the team that i worked on we mostly focused on managed kubernetes we we you know we're using eks mostly uh before that um it it was a that was a long road um we we definitely identified kubernetes as the way we wanted to go because it allowed us a common language to use between our development teams and our systems teams and you know it wasn't like uh every app wasn't the snowflake or this pet type of configuration that we'd set up so we i liked that we kind of came together as you know collective teams and decided you know kubernetes is our future we started with uh the cops project and you know tried spinning our own clusters up that way but um when you do that uh the the open source community is great but it makes it harder for big organizations to you know call up and uh and ask questions and kind of get engaged uh again community is great but it doesn't give you that like support contract and that's what big enterprises look for so we went with uh rancher for a bit and using that as well as terraform to stand up you'll see you'll see why i'm at hashicorp soon i love terraform um but we used terraform and rancher to stand up and manage our clusters so we were hosting our own control planes and hosting our own uh worker nodes and that worked really well for a while uh but for for anyone that's that's run a control plane in kubernetes the xcd data store if you're running that through any abstraction you might have some problems from time to time just as life happens and comes at you so we saw some issues there and just being able to kind of uh delegate away all of those fcd and control plane problems and scaling issues to to amazon was was a very good move uh for our teams and so we really enjoyed that but uh that's mostly what we would use and then that would give us the time you know it was it was you know a little bit behind the current releases that were out and so taking that time to be aware of you know like in 116 there were a lot of api changes taking the time to to work through those and have the time to adjust was really nice and so again you know that's that's that's why i'm excited about that one year this time cycle for for some of the bigger end users and people that aren't able to quickly change everything over just because of the scale and the size that they're working at yeah i'll just echo some of your points there but every production outreach i've ever had with a kubernetes cluster has either been cube dns or an ncd so if you can if you're fortunate enough to be able to use one of the managed services then definitely you should be using that and i've got this cluster api which maybe we'll we'll talk about later so back on track now right i'm going to try not to segue far too much but uh the next major theme on the kubernetes release blog was the storage capacity tracking what's that so i'm gonna lump these to these uh these oh i'm reading ahead um so there were a lot of really interesting features added you know in alpha state in this release so there is storage capacity tracking generic ephemeral volumes and csi volume health monitoring all of this really being that you know the expectation of kubernetes clusters and and those use cases were kind of assuming that you had infinite availability to um to storage uh uh or just volumes that you would actually use to store things up so just overall storage the thought was that that was infinite um now we're able to with the csi in place container storage interface we have the ability to be a little bit more intelligent at looking at you know they're like i'll just talk about amazon just because that's where i've gotten all that's where i have a lot of experience so when you take a look at you know ebs volumes versus efs versus kind of all those storage classes that you have um you you also have like the i nodes and those have a really fast uh local disk storage the nvie i believe and um so you're not able to really leverage that uh those volumes because you know you're either dealing with block storage you're not really too focused on that local storage on that machine i can't remember which kubernetes release it was but i know that that you know uh once that was kind of pointing out to the community they said hey can we just make use of the disk that we actually have hosting on these machines and so that was added in so this allows us to be a little bit more direct when it comes to how do we divvy up our you know storage classes and how do we how do we actually look at storage so these these three features have really set the uh set us up uh for success in the future for uh identifying you know different types of storage classes instead of just you know throwing an ebs volume on everything which is which is quite nice uh the the health monitoring is also really good too because then you can actually see when that volume gets mounted to that machine and so on and so forth so a little bit better reporting on that front too which is which is quite helpful because uh you know troubleshooting things in the in the uh troubleshooting things with that a lot of information becomes very difficult oh yeah definitely that's for sure so those three features then are the storage capacity tracking which exposes metrics about the utilization of the different csi and block devices etc that we have i think the feature you mentioned there was that the local provisioner from 1 17 i think that allowed you to actually access the bare metal disc exactly exactly uh and then the second storage one there is the generic ephemeral volumes what what does that allow me to do then from a kubernetes point of view so what that allows you to do is kind of have like a working desk so if you have something that you don't want you know it is ephemeral that you don't need around for a long time and you don't want to use your that local machines uh you know at volume at that point in time you can kind of give it a different disc than the root disc and then you know do whatever operations you might need to with that and then kind of throw it away which is which is kind of nice so it's a scratch pad essentially oh okay so if i have a kubernetes deployment and i want to leverage some of the csi implementations but for ephemeral things i can't even think like say i wanted to store my logs but not on the bare metal that's but on csi implementation then i could request an ephemeral volume for all that provider exactly yeah and you might have different needs you know if you might need something with higher iops or something like that too you can get that that quick you know burst ability have it be ephemeral and then and chuck it once you're done with it okay i i think i get that now the the last one that we were talking about there was this csi volume health monitoring but how does that who can then to those last two features and what what am i getting on top of there so what that will tell you is is so the storage capacity tracking that just kind of lets you know like hey we're like you're really leveraging uh this volume over here it might not make sense because you have a lot of disk pressure might you know you might not want to put things over there uh the csi volume health monitoring what that really allows you to do is just kind of get a sense into the events that are going on so if you have you know a ceph cluster if you have ebs volumes you can kind of keep track of where they're at and how they're doing so if you have you know a machine go offline you're going to want to know about that if for for anyone that's used ebs volumes those are locked to one availability zone and so that kind of makes things difficult when you want to have you know say a uh db cluster and if that goes down you know and you need to bring it up into another az you you can't you have to stay in that same az you can fix that with efs which is multi-az but you're still going to want to have metrics on getting that rescheduled i've had nightmares and real life experiences with evs volumes and it taking upwards of like four or five minutes to rejoin a machine and to reconnect and so having a little bit more tooling on you know where is this volume how is this you know what's going on and having some more error codes to source from uh make that a little bit easier experience too so that you can kind of tune and and set up the cluster exactly the way that you'd want when it comes to volumes nice so 119 is a pretty good release then for as far as staple applications and storage go then oh absolutely uh do you have any strong opinions on running stateful applications on kubernetes uh i so so abs i i've been there i feel the pain when it comes to supporting you know stateful applications i've been lucky enough that a lot of the applications i've worked with have been stateless so not too much of a concern i've run though conversely i've run elasticsearch i've run mongodb um i've i i've played around with running postgres you know i i'll typically pull down rds or something like that but uh definitely worked with postgres in a helm chart and uh it's i really like that you're able to do things like that but it just is it's you know it's it's something that you have to think about is are you okay with that um because you will run into problems you absolutely will no matter how much tooling is available to you managed services are always easier to a set up and to be or be you know have managed because they're managed services and you have support you can call people and work with them on that front but you know it's it's it's an extra layer that you really have to think about uh you have to think about if that were to go down you have to think about your backup process you have to think about and do disaster recovery scenarios so if you're able to do that i'm all for it i think it makes a lot of sense and it's a lot of fun because you don't normally get that level of insight into the types of workloads and applications that you support um but at the same point in time that is a lot of you know that can be a very big time sync and if you're you know if you're a smaller team that becomes a little harder to do so you know smaller teams or teams that support a lot of applications i'd definitely lean towards suggesting a managed service otherwise uh stateful sets let's let's go let's learn some things because that's that can be beneficial too that's its own uh that's its own uh payment kind of kind of point yeah i think either you've been extremely lucky in your career or i'm a bit of a masochist but you all the innovators you mentioned there are distributed by the poll the mongodb you know already handles you know working across multiple nodes whereas i seem to be very unlucky in my career and work with databases that did not support that which makes my life a lot more painful difficult and fun let's go with fun i i i maybe at some point in time we can uh we can do some live coding and get like a cockroach db cluster set up and something that's supposed to be easy and support sql native calls maybe we can we can find some milligrams between us at some point well yes definitely i'd be up for that session for sure because i i mean i work for a bare metal company which means there are no managed services so even though i've never worked with a distributed database previously i'm now working for a company it doesn't offer managed services so i maybe his masochism is my [Laughter] okay so the volumes are cool i really love that um i think anything that makes stateful applications on kubernetes easier for people to consume is fantastic now the next one on this list that we have shocked me a little bit and i don't know if i'm just being really naive or not but england graduates to general availability was that not generally available right that's i'm not kidding you despite being you know this is a true you know taxicab confession uh yeah i thought the same thing walking into the 119 release i had had a double take the first time i saw i was like wait a minute what uh it was a a gmail beta you know type uh we're getting rid of the beta it's like wait a minute that was beta uh type moment for sure um when i i can't remember the and forgive me you know i'm still still booting up with some caffeine here but um i can't remember which uh kubernetes enhancement proposal it was but um there was one about kind of reevaluating and looking at that alpha beta ga type classifications because we would have the situation where something would go into it's a chicken and egg problem when you have something going to alpha you've got into kubernetes that's fantastic but most of the time it's hidden behind the feature flag that you'll have to enable or you know pass the cubelet and then that will get supported um and with anything alpha you know it doesn't really scream production no you know nor should you in many cases unless you're just you're knowingly you've accepted the risks and you're testing out a new feature never turn on alpha that's my my recommendation i think the uh any lawyer is going to back that up too uh but uh when so we get into this situation where all of these new alpha features are really cool like look at the ones that we're talking about they sound fantastic but if you don't have that beta or higher type of certainty that it's actually going to go live at some point in time you don't have a lot of certainty and that's what we like when we're working with infrastructure so um the the language was changed there and just kind of that that process because a lot of people would come forward and say hey i have this thing in alpha can we just move it to beta because that will get people to use it people are you know typically not not uh jumping to use alpha features so that makes it hard to move to that beta stage even if there is that level of interest it can still be pretty difficult so the language and just kind of that overall process was changed again i can't remember the kept uh that that one was around but that's why the ingress you know working as it was uh it just needed some final things to push it across the finish line so that we could deem it as stable and ga and then you know start start working with the community on more features so that's that's what i'm pretty excited about that's one of my personal favorites along with the longer support time is the uh is that ingress finally moves to ga woo i think it's one of those one of those things that because i'm going to get this wrong we are just so conditioned i feel as users of kubernetes to v1 beta 1. that to me that was ga like because we just had all those apis for so long and of course now we've got v1 but i just always assumed that ingress was one of those one of those objects that was just available i mean every kubernetes cluster i've had for years now has used english and i never once thought for a second it was something that was in beta so same and it worked so well it's it's worked so well so it's you know just like gmail did for 11 years i think it was before it when it went ga you know it's it did everything you needed it too yeah and at least yeah you're right it's always worked so it's fine it's good and i think yeah it's it's it's nice for people to understand how that new feature thing works from kept to alphabet and to ga and i think you kind of covered that in a complete journey of a cat which is nice and cool and angus is the ga you can start using it i guess you know if you weren't already i think they did they changed the the name space as well was that this release of the previous one i can't remember now i think that's when i first realized that things were as i always expect them to be cool uh so we've got two more we've got three more but there's two more i think we're probably going to grip together now the next two major themes are structured logging and the k log methods so structured blogging is going to be fantastic uh because and well it is uh it's really fantastic because it allows us now we to get standard logs out of kubernetes um we could do things with standard out and standard error and you know that mean it made it easy when we had a log aggregation layer in our stack like elasticsearch or something else um and then you know but it did become kind of a pain point when we started indexing those logs and because you were never really you got logs but they just weren't in you know a similar format so that indexing became difficult and then sourcing what data you actually needed to see became difficult um that you know mutating those logs downstream before you put them into your login cluster that that becomes an issue so finally you know uh the there's there's a feature now where we actually have everything time stamped and then you have like a logical standard out and standard error to actually source from um there's we'll kind of go through a quick demo on that one and here in a little bit but you'll just be able to see those lines are now different because they start with that time stamp and it's just in a standard format so even if you you know advise your teams to log in json or anything else you're if you're just doing you know line by line type logging that's going to be consistent now and you don't need to worry so much about that or making crazy regular expressions to deal that and to capture those you know i i'm sure everyone agrees with me that there's better time better better time spent than on regular expressions you know apologies to those of you that really enjoy it i have wasted so much of my life writing rejects and grock parsers for long stash and telegraph and all these other tools but just admitting it as json is such it just makes everything so much easier i think the way that we consume all of these logs and like i said we're going to do a demo in a few minutes people's hopefully just see right away the power that we get the flexibility that we're going to get from this approach i had the spoilers i was not able to get the the thing i really like i just haven't been able to replicate on that front with the logging is that output in json but so i'll show you what you know just just regular uh standard out and error looks like but uh being able to once you're able to tap into that json this feature you know both those features are in alpha now but being able to work with that and see that get to ga is so exciting because you know like like david said uh lost years of my life and uh there's i don't see any right now but at least like 15 gray hairs that have that will be coming out from the regular expressions i'm sure yeah and i i don't want to uh make any naive assumptions but you know it's like it's more formality right for such a blogging to go in as an alpha just because it's a new feature i don't really see what would hold that up from graduating to better and stable other than formal field possession exactly it's just kind of that that steep time that you know just time to work with the community and then that new k-log method they've given you know the community the the project that with that feature you're able to kind of take your time and then kind of lean everything over into that new logging library such that you can really leverage it um you know despite despite what's uh what's coming out logs as it is right now sweet awesome i'm looking forward to playing with that all right we've got one more thing on our super duper list which is the cubelet client cls certificate rotation so this one is really cool i really should have made my list i don't know why i haven't talked more about it uh but uh i think mostly because it's it's been something that we've been used to and that has kind of been done uh you know behind the scenes and that that is the the cubelet client tls certificate rotation um so that that did happen in the past but that was happening with an outside of the cluster mechanism so with this feature that's gone ga that finally is brought into the cluster so that as needed the control plane can kind of roll those client certificates and it's just not something that you need to think about so when the cubelet calls in and it needs that new one it's just a lot more seamless you reduce that level of latency and uh and you know everything just works trademark there you heard it from taylor himself everything is just going to work just like i have to go i'll see you later so uh so what we've covered we'll just share my screen here is pretty much the majority of this article which is the really something law that just tells you what's new at the bottom we do have a couple of things that we haven't touched on and some of them are quite important so the first one is is that the set comp integration has now graduated to stable and assuming things go well we will take a look at that and how that works i'm moving away from the old annotation approach to the new actual quad spec level stuff which is really cool uh and i'm not sure if there's anything else here that you want to quickly run over like the run multiple scheduling profiles i mean that sounds kind of important but i don't think i fully understand it is that something you're comfortable talking about i think with most of these so i'm still honestly i'm still working through a lot of these myself and just kind of trying to find uh you know i'm having a lot of aha moments after um after the release uh getting to kind of see these three things worked through see the tests that you know they've been put through in prow um but uh it's still working on quite a quite a few of these and and how they apply to my workloads but uh yeah so none that i could talk to right now is the tlbr but uh very excited too yeah well i mean all of these are links to the caps and issues that people can obviously go if anything there it makes you piques your interest and feel free to go click on it take a look at it the other thing we are going to take a look at today however is the immutable secrets and config maps which hopefully we can get that working and show you how cool that is or painful depending on your program and another one i really like which is let's see that's a major change but i think it's been graduating over the last couple of releases it's a no topology manager which actually allows you to do rack and data center aware service reading which is really really cool too and maybe something that we will touch on another date okay i think it's time to get hands on you feeling brave ah always all right so i'm not sure i'm not sure if that's a good thing or a character flaw but it's a good thing it's a good thing okay so we're gonna do the immutable secrets and contact maps psychology a and structured blogging do you have a preference of what you would like to start with uh i could kick it off with yes if if you wouldn't mind grabbing the immutable secrets and stuff com i can kick it off kind of easy breezy with some structured logging for sure all right let's do it let me share my screen and click the right buttons and then we should be there all right so everyone should see my terminal let me get that at a decent size for everyone so uh so i have spun up actually let me let me get rid of this all right so uh i'd say like i'm gonna delete things with you live uh okay so uh the kind project that i talked about a little bit earlier is really fantastic because it allows for you to uh spin up a local kubernetes cluster so i'm actually gonna cat out this this configuration so i was working a little bit earlier with the sec comp which i've you know happily happily uh uh hot potatoed over to david for uh for one of our next demos but you can see that you're i'm able to specify the 119 release here and then working with some of those those profiles you can see that there are different arguments and ways in which you can define this when you're spinning up your your kubernetes cluster please note that up here in a roll that's actually going to be the different machines you know virtual machines here that i'm going to be spinning up and working with so a huge thank you to the the team that manages kind uh it's it really makes it easy when you're working with uh with local workloads in addition to you know things like mini cube and docker for mac uh windows linux and spinning up those kubernetes cluster but the benefit here is it makes it very easy uh to go ahead and grab the latest version there so so i've deleted that out uh i'm going to do kind and let's uh create a cluster i also like the tooling just because it's you know all the command line items just make a lot of sense i also do very much like the emojis you can see here this is an image image you got machines here with with the nodes the package i really like that um as you uh configuration you got a scroll uh control plane you have game controller fantastic um but i think i think i figured out most of these but i'm not 100 sure but we'll see we'll see how well my my skills of relation are as we go through this but the good thing about this too is that it doesn't really take that much time i think it's the control plane step and when it joins the worker nodes to that control plane that's like when you're waiting for 20 30 seconds it's it's typically not that long so hopefully you don't have to eat my shoe uh after after uh saying that but uh so yeah you have uh cni storage class uh this worker nodes just takes just a little bit of time um but then your and then it writes that cube config uh to your local file system and you don't need to worry about that either so once once this is done i'll kind of just show you that we have a working cluster and then i'm just going to tail the logs to cube or core dns to show you what those standard logs look like cool so we have a cluster um let me uh so i a couple things for those of you following along at home i've aliased the cube control just to k so if you see me type in k i'll i might bounce back and forth depending on the commands but that's that's my alias and setup i also use two two tools uh one is uh cube context and uh cube name space and that just makes a little bit easier for me to switch between clusters because you know i'll typically manage quite a few so if we go ahead and we take a look at the nodes kind switches over that context to the cluster that you just spun up so you don't need to switch to it which is kind of nice so we can see that we have four machines here one is the control plane and then we have these worker machines here um if i jump into the oops if i jump into cube system here make a little bit easier for me to get these pods and see what's up so we can see we have our whole control plane here uh and say i want to go ahead and take a look at this uh i don't want to define that get out of here um i want to take a look at the logs here and follow along uh we can see that you know here are some of the errors uh that's not so hot but you know i'm assuming everything's working working now uh we can see that we have time stamps and then we have these you know just the regular body of the message beforehand you know you would get just separate pieces of information and data but if we let's take a look at another pod let's take a look at let's take a look at this one but you can see here you know now everything is standard in the fact that we can always rely on that time stamp being first and then getting that that you know downstream data in the body of that message later again i apologize was not able to get the um the json view uh up for for this i think there's a couple uh flags i needed to double check and take a look at my setup but you are able to export this as json if you specify the right commands and that is going to be that's going to obviously be a lot easier to kind of look through and then filter on as you're trying to tell logs and things like that so really excited to see how this evolves over time and as we're able to you know filter on like i only want to see error logs and um you know the biggest one being that it's so difficult to kind of link together if you have a deployment that's three pods to kind of tie all those logs together so that you get a comprehensive view as to what's going on without having to stand up your own login cluster so that's uh that's structured logging nice and because i am feeling brave and i have been working on the json stuff i'm going to give that goal uh and i'll i'm using a mini cube instead of time oh and i'm i will i'm not feeling good because i can't even spell i will feel brave and delete yes uh and so the way this would work i feel particularly brave hold on let's jump over here so i have a i guess it may be too brave but i've been working on a branch of many cubes and to make this a bit easier so let's let's look at what this would work let's let's look at how this would work normally so is that begging so many cable legends to pass dash dash extra config which allows you to specify extra parameters that can go to each of the kubernetes components but what i can do here is i can say that i want to pass something to the cubelet and the key that i want to modify is going to be the logging format and i can specify that as json like so unfortunately i could do that for every component requires doing this and saying hey let's do the scheduler 2. and then i'd have to do what else there would be the api server that would be the proxy there would be core dns like that lane would get super painful and i'm a very lazy developer so what i started doing with this ability where we can just specify login format equals json and you'll see that i've added some crappy characters to the end because i actually don't know how to write tests for mini cubes so i wanted to make sure it didn't work as well um and that was the way that i did it so you can see here that it just says hey this is not a supported login format so let's just do that again with json now many cubes should take i mean i'll look back but i'm just i'm gonna assume it's gonna be okay and it should just take oh it's not okay instead of delete maybe i had one cluster and a half set up state there right if that doesn't work outside they will do it but uh it should just take roughly about one minute maybe a little bit less together as a working note and what we'll see following on that we can already see the parameters being added now to the different components so the schedule of the controller manager and the api server and we should be able to pull out those logs much in the same way the taylor did only hours will be in a json representation the cool thing about that is it's going to be really easy for that to be parsed by log stash unless you search vector or whatever tooling you have and your step cool that doesn't crash this time so i'm happy so if i run i do not have the fancy tool like you do cube name but i could do k dash n cube system and i'm going to pull out the api server logs and when i do that it's not json ah so that's my and you know stupid that just means that my fork didn't work so let's just quickly do this all the way there and i'll just do the api server that was really unfortunate i was so confident i swear it's uh it's as soon as you do a live demo that's uh there's something that happens you lose your magical powers don't know what it is but yeah being alone you know and just that solitude that focus uh really it really brings it out of the demo yeah i don't know what i did wrong there because i have tested this but i build their own api server and i don't enable it on the api server oh well it doesn't matter so what i did there is api server now this doesn't work again that means the api server just ignore that oh it did okay so you can see here um we don't actually have json for every single lane and if i just pick that on a keynote paper jq we have this one it's easy to parse i can use other tools with a much good and that's it structured login is not one of those things that has a substantial amount of change i think most of the changes came from reusing the different keylog methods within the instrumentation um but nice good i'm not going to script anymore i'm not brave i'm not not not at half past six on a monday okay so starting logan very cool we like it and we're looking forward to see tools take advantage of that and store logs with good debugging stuff what did we say was next what is next i think it was the i'm not sure if you want to save the um oh uh this is this one should be pretty easy immutable secrets and config maps yes right okay so let's pop open vs code and let's deploy i'm not going to but we're going to config map and we'll call this rpfm and we can just have any arbitrary value like so now i guess we should just show maybe talk about how this works currently without making any modifications because i don't think people really understand what happens and why this is a bit of a pinpoint cool if we do apply this then we can do here i'm in bash and then i'm not getting my elites uh get configmat rtfn if i modify this value but i didn't show the value you can see here key is 1545. now if i change this and reapply it and i run get we have this updated value now that can be considered good or bad depending on what you're doing with your kubernetes cluster so i'll speak from my own experience and then maybe you want to chip in with how you feel i don't think of i i don't know both of the same opinion but there's this weird thing that everyone on the kubernetes journey has to deal with at some point and that's where you make a modification to a contact map you have running pods inside of your cluster and there's actually no notification or event that tells that pod that it needs to restart because the configuration changed and this is so common that there are a lot of helm charts now that will actually put an annotation on the pod spec with the sha of the conflict to force that thing change but the other school i thought is the config maps shouldn't change anyway and that you should always deploy a new convent map and then with a new deployment that consumes it or an updated deployment that consumes it is that kind of a lame reviewed experience as well oh absolutely it's it's when you have something especially when you follow like you know a get ops type method methodology where what you see in that repository or like when you set up a cluster an app something you you have to have so in my opinion you have to have something that is static so that you can reproduce it if you have a config map that's mutating or changing secrets that are mutating or changing you don't get that level of certainty and so that makes it kind of difficult to troubleshoot to reproduce issues that happen things of that nature and this also say this were you know a bitcoin wallet and uh and i found out what david's was i could quickly change that and then you know uh we could we could use uh his money instead of mine and you wouldn't want that would you no so i guess yeah and just to kind of really cement why this was a problem is that different and for some of your pods in your deployment if you get 20 pods and some of them restart they could be considered an older version of the configuration some could be modifying the new version and if you've actually applied three four or five changes to your contract map for whatever crazy thing you're doing all of your pods could that they would be running a different configuration really weird we had a comment i think someone noticed the error with my manicure by the way it was that i possibly wasn't actually using my locally compiled one but i'm not going back to it uh i'll hopefully get that pull request merged and everyone else can play with it so 119 gives us this new thing and it should i'm going to say it should be as simple as adding a new key to the config map that says this is immutable okay so we now have that configured if i just pull it back down you know i've never actually modified and mutable conflict map to be immutable so let's see if this breaks or not but you can see here we have the value immutable true and the value is 678. now if i've done this correctly and i decide that this should now be abc one two three when i apply this we should get another message that says you cannot modify this conflict ah it works yes so it's one of those really small changes the api surface is really small what immutable true and it's done but i think the quality of life for people that are consuming and using kubernetes and the deterministic nature of now that a deployment in a conflict map will always be in sync is really really important to people at least i i think this would be a very welcomed addition with the api for sure absolutely agree and and secrets are just as easy too all you need to add is that immutable true flag and you're good to go but uh definitely agree with david on that front it's uh even figuring out you know what what is what is a secret in kubernetes what is you know worthy of putting into config map and at the end of the day that's typically like anything you're okay with seeing in plain text um is usually a pretty good uh smell smell test you know before you actually put it in there as well so you know credit cards not a good config map entry to put in there no all right we got uh two follow-ups in the chat there so the first one being uh another big issue is if you have a conflict map managed in a separate repository it's a pain because no kubernetes native event yeah so if that's just reaffirming that deployment situation and and i think the use case here is if you've got like a platform get ups repository that provides those environment information and then your applications are deployed separately there's no way to connect the dots between that conflict map changing and what's in the cluster yeah as a huge problem immutable conflict maps just remove that because you always have to deploy a new one and update the surrounding thing so yeah i think it's cool the other question was can we delete can we delete an immutable conflict map uh yes he says confidently i did do this earlier so it's gone and there's nothing there to stop you deleting that there's no protections from there's nothing there to stop you deleting something that's going to be consumed by it so yeah if that's a challenge you're going to end up with pods and a fail to restart um container create loop if you delete one that is needed but at least you can be sure the content's not very good yes absolutely there there was one edit too because um even though like we had said before david pointed out the fact that when you change a config map that doesn't automatically restart all of your pods you know unless you have some you know some some out-of-band tooling that handles that but kubernetes native does not give you that that type of experience however kubernetes keeps track of you know hey this config map was updated right because it has to store that state and so when you mark it as immutable uh kubernetes gets that very you know and i believe in most cases very small performance gain where it doesn't keep checking to see if that config map has been updated because it knows it can't be uh so there's there's also that like hidden benefit as well uh you know at the end of the day i'm not sure how much uh how much performance it's going to net you but uh just something that's cool might be a good uh trivia question or something to pay attention to with that all the cloud trivia that's that that we've seen this yeah so that's a really actually good point because what you've just said has now put in a bit of ambiguity into the problem domain that i was mentioning so i think we should let's just clear this up now the conflict maps are and if i get this wrong please correct me whenever exposes environment variables or fails will fails and environment variables are updated within the pod the challenge is the process running normally doesn't know to take advantage of the reload which is why you end up with multiple processes consuming old information it's not that kubernetes doesn't update the fail or the environment variable that's right yeah exactly exactly cool and i think what we should just do now is show off one other thing because i'm sure there'll be a question on it later but what if i deploy this is immutable and want to make it mutable i was going to say it unimitable and i would have just left it so we now have an immutable contact map and if i remember correctly you cannot then make it mutable this will not work okay so if you make a compact map immutable there is no undoing it to do online or hot updates you always have to create a new config map but i can't believe so let's get rid of it that reminds me of uh slack channels how you can make it private but you can't then turn a private channel back into a public one which also makes sense yeah i guess so yeah i don't think there's anything else to cover on the immutability here i think we've pretty much covered every potential education question so at least we have to move on to the difficult bit so we're going to talk about the let's close that the promotion of the setcomp api uh was the general available yeah i think it was uh yes i cheated i have downloaded a set comp profile i am not brave enough to try and put that together during a live stream uh and this is for nginx what is really cool uh is if you deploy the setpump operator this profile is actually provided for you i'm not going to show the setcomp operator today there's a stream on that last week feel free to check it out what we're going to look at today is just the raw kubernetes and working with the api directory so what does that mean we need we need a deployment let's see that's vs code extension where you never have to actually type the yaml i don't even know if i could type a deployment yeah anymore wow so i'm just noticing i need to add that and save myself hours of time thank you dude so let's remove the resources i don't need the port but i'm just because i'll add the port now what happens here is i can deploy this engine x resource it's like any normal resource which i mean i don't get anything wrong oh i'm not playing the json because i call this the plugin there we go which means i can now do a quick control get parts let's just get that second it's gonna be pulling down the engine x image run it on a watch of course it's running and now i can do exec engine x not my real shell so i can't get all complete and let's hit the next container and i can drop back in nothing exciting there all standard stuff however is a certain attack vector without specifying a second profile and that is if a malicious actor somehow manages to get within the process can they execute commands that we do not want them to execute like me exit me getting inside of a pod seems harmless but you know the ability to get inside of that name space could be potentially dangerous right so we can use the setcomprofile here which lists all of the syscalls that nginx is expected to make and restricts it so what we have to do is copy this and then if i do many cubette so you have to be on each of the hosts or the nodes and you have to put nfl i'm going to have to convert and we're going to add a directory to the public configuration so that is where all the contact lives we can create a new deck create a new directory called segcom profiles and i need to do an app update and install them because i'm not going to have it i just realized magic stuff happens saying the internet's not too bad and we go inside setcomp profiles now from here i can create enginex dot json hit that and leave it alone so this would also be part of your configuration management setup you might have terraform write this file or salt stack or puppet whatever you're using put all of your profiles into a single location on each of your workers so they can be consumed by the pots or you can use the second operator which does all that work um now we want to restrict the syscalls that engine x can make and to do that these changes to the api leverage the security contact api so this already exists allows you to do things like what force the uid the gid i think there's a few other things but now we can specify second profile and we have a type so this is where i'm going to start getting into territory that i'm not entirely comfortable with but i'll try and remember it and there are localhost types and container runtime type i think it's run pain default um the documentation is going to have that covered but this is a profile that ships with your cri implementation which tries to remove some of the more dangerous syscalls like privilege escalation and such but it's not really specific to applications you're going to run for that we have to tell to use the local host which is going to use the profiles in the directory we just provisioned and then we can tell it what profile we want to use now assuming i've not messed this up which is extremely likely i should be able to do profiles engine x dot json and that path is just uh profiles here the directory that i created and the nginx.json here and we can apply that so i'll drop out of my minicube we'll do it here ssh apply and reader the deployment no we will oh drop as many cubes now i can do it okay now if i've not messed this up i should be able to run get pods and see the pods running good and we can confirm that it and has we expect so describe pods pod name and then we'll just grab for second you can see here we're running this nginx profile and what that means is i'm going to use the pod name again as i try and exec and say that this it feels good so what happened there is this syscall get p process group i think that's a cisco failed because our second profile doesn't allow engine x to run that cisco and i can now no longer get in the container the good thing about that is if someone were able to exploit an nginx vulnerability or cve to get inside of that container the chances of them being able to execute any commands that would potentially allow them to get back down to the host should hopefully be nothing but hopefully uh i think that's it like um the api surface is nice and small unfortunately right in the second profile is really difficult but at google try and find ones other people have written and but the application from the kubernetes point of view could not be simpler you know a couple of lines on the security context and you're good to go is there anything you want to add to that no i think just very much agreeing with you that the writing a set comp profile is is you know at least right now difficult i definitely want to learn more on that front as well i think i'd feel more comfortable writing an autobiography before.com profile as it stands today but uh no absolutely it's uh 119 is fun uh go check it out uh if you have any questions about features you know please feel free to reach out um if there's any you know demos or you know you feel like there's a blog entry that's needed or tutorials things like that hit us up um you know and it's it's kind of fun to see what people are using to talk through those work streams and those workflows and just get a sense as to what everybody's interested in that's uh let's have a conversation awesome well thank you so much for joining me today this is really good to just kind of sit there and go through some of the new features you know understand how to work and think instead a little bit more so thank you and thanks for having me no this is a blast uh uh hope you all have a wonderful rest of your monday and uh yeah thanks thanks for hanging out with us all right thanks again taylor i'll speak to you soon bye thank you you

Channel: Rawkode Academy

Views: 116

Keywords: Kubernetes, Cloud Native

Id: OReOSuX-3YE

Length: 73min 25sec (4405 seconds)

Published: Mon Sep 14 2020